Python Machine Learning Tutorial, Scikit-Learn: Wine Snob Edition

In this end-to-end Python machine learning tutorial, you’ll learn how to use Scikit-Learn to build and tune a supervised learning model!

We’ll be training and tuning a random forest for wine quality (as judged by wine snobs experts) based on traits like acidity, residual sugar, and alcohol concentration.

Before we start, we should state that this guide is meant for beginners who are interested in applied machine learning.

Our goal is introduce you to one of the most flexible and useful libraries for machine learning in Python. We’ll skip the theory and math in this tutorial, but we’ll still recommend great resources for learning those.

Before we start…

Recommended Prerequisites

The recommended prerequisites for this guide are at least basic Python programming skills. To move quickly, we’ll assume you have this background.

Why Scikit-Learn for machine learning?

Scikit-Learn, also known as sklearn, is Python’s premier general-purpose machine learning library. While you’ll find other packages that do better at certain tasks, Scikit-Learn’s versatility makes it the best starting place for most ML problems.

It’s also a fantastic library for beginners because it offers a high-level interface for many tasks (e.g. preprocessing data, cross-validation, etc.). This allows you to better practice the entire machine learning workflow and understand the big picture.

WTF is machine learning?

• Ahem… maybe this is a better place to start instead.

What this guide is not:

This is not a complete course on machine learning. Machine learning requires the practitioner to make dozens of decisions throughout the entire modeling process, and we won’t cover all of those nuances.

Instead, this is a tutorial that will take you from zero to your first Python machine learning model with as little headache as possible!

If you’re interested in mastering the theory behind machine learning, then we recommend our free guide:

• How to Learn Machine Learning, The Self-Starter Way

In addition, we also won’t be covering exploratory data analysis in much detail, which is a vital part of real-world machine learning. We’ll leave that for a separate guide.

A quick tip before we begin:

This tutorial is designed to be streamlined, and it won’t cover any one topic in too much detail. It may be helpful to have the Scikit-Learn documentation open beside you as a supplemental reference.

Python Machine Learning Tutorial Contents

Here are the steps for building your first random forest model using Scikit-Learn:

1. Set up your environment.
2. Import libraries and modules.
3. Load red wine data.
4. Split data into training and test sets.
5. Declare data preprocessing steps.
6. Declare hyperparameters to tune.
7. Tune model using cross-validation pipeline.
8. Refit on the entire training set.
9. Evaluate model pipeline on test data.
10. Save model for further use.

---

Step 1: Set up your environment.

First, grab a nice glass of wine.

Next, make sure the following are installed on your computer:

• Python 3+
• NumPy
• Pandas
• Scikit-Learn (a.k.a. sklearn)

We strongly recommend installing Python through Anaconda (installation guide). It comes with all of the above packages already installed.

If you need to update any of the packages, it’s as easy as typing  $ conda update <package> from your command line program (Terminal in Mac).

You can confirm Scikit-Learn was installed properly:

Great, now let’s start a new file and name it sklearn_ml_example.py.

Step 2: Import libraries and modules.

To begin, let’s import numpy, which provides support for more efficient numerical computation:

Next, we’ll import Pandas, a convenient library that supports dataframes . Pandas is technically optional because Scikit-Learn can handle numerical matrices directly, but it’ll make our lives easier:

Now it’s time to start importing functions for machine learning. The first one will be the train_test_split() function from the model_selection module. As its name implies, this module contains many utilities that will help us choose between models.

Next, we’ll import the entire preprocessing module. This contains utilities for scaling, transforming, and wrangling data.

Next, let’s import the families of models we’ll need… wait, did you just say “families?”

What’s the difference between model “families” and actual models?

A “family” of models are broad types of models, such as random forests, SVM’s, linear regression models, etc. Within each family of models, you’ll get an actual model after you fit and tune its parameters to the data.

*Tip: Don’t worry too much about this for now… It will make more sense once we get to Step 7.

We can import the random forest family like so:

For the scope of this tutorial, we’ll only focus on training a random forest and tuning its parameters. We’ll have another detailed tutorial for how to choose between model families.

For now, let’s move on to importing the tools to help us perform cross-validation.

Next, let’s import some metrics we can use to evaluate our model performance later.

And finally, we’ll import a way to persist our model for future use.

Joblib is an alternative to Python’s pickle package, and we’ll use it because it’s more efficient for storing large numpy arrays.

Phew! That was a lot. Don’t worry, we’ll cover each function in detail once we get to it. Let’s first take a quick sip of wine and toast to our progress… cheers!

Step 3: Load red wine data.

Alright, now we’re ready to load our data set. The Pandas library that we imported is loaded with a whole suite of helpful import/output tools.

You can read data from CSV, Excel, SQL, SAS, and many other data formats. Here’s a list of all the Pandas IO tools.

The convenient tool we’ll use today is the read_csv() function. Using this function, we can load any CSV file, even from a remote URL!

Now let’s take a look at the first 5 rows of data:

Crap… that looks really messy. Upon further inspection, it looks like the CSV file is actually using semicolons to separate the data. That’s annoying, but easy to fix:

Great, that’s much nicer. Now, let’s take a look at the data.

We have 1,599 samples and 12 features, including our target feature. We can easily print some summary statistics.

Here’s the list of all the features:

• quality (target)
• fixed acidity
• volatile acidity
• citric acid
• residual sugar
• chlorides
• free sulfur dioxide
• total sulfur dioxide
• density
• pH
• sulphates
• alcohol

All of the features are numeric, which is convenient. However, they have some very different scales, so let’s make a mental note to standardize the data later.

As a reminder, for this tutorial, we’re cutting out a lot of exploratory data analysis we’d typically recommend.

For now, let’s move on to splitting the data.

Step 4: Split data into training and test sets.

Splitting the data into training and test sets at the beginning of your modeling workflow is crucial for getting a realistic estimate of your model’s performance.

First, let’s separate our target (y) features from our input (X) features:

This allows us to take advantage of Scikit-Learn’s useful train_test_split function:

As you can see, we’ll set aside 20% of the data as a test set for evaluating our model. We also set an arbitrary “random state” (a.k.a. seed) so that we can reproduce our results.

Finally, it’s good practice to stratify your sample by the target variable. This will ensure your training set looks similar to your test set, making your evaluation metrics more reliable.

Step 5: Declare data preprocessing steps.

Remember, in Step 3, we made the mental note to standardize our features because they were on different scales.

WTF is standardization?

Standardization is the process of subtracting the means from each feature and then dividing by the feature standard deviations.

Standardization is a common requirement for machine learning tasks. Many algorithms assume that all features are centered around zero and have approximately the same variance.

First, here’s some code that we won’t use…

Scikit-Learn makes data preprocessing a breeze. For example, it’s pretty easy to simply scale a dataset:

You can confirm that the scaled dataset is indeed centered at zero, with unit variance:

*Side Note: If you are getting values like 1.16664562e-16 it has to do with how your system handles rounding and precision. To get rid of the scientific notation and simply print 0, you can run the code np.set_printoptions(suppress=True) after you import the NumPy library (from Step 2).

Great, but why did we say that we won’t use this code?

The reason is that we won’t be able to perform the exact same transformation on the test set.

Sure, we can still scale the test set separately, but we won’t be using the same means and standard deviations as we used to transform the training set.

In other words, that means it wouldn’t be a fair representation of how the model pipeline, include the preprocessing steps, would perform on brand new data.

Now, here’s the preprocessing code we will use…

So instead of directly invoking the scale function, we’ll be using a feature in Scikit-Learn called the Transformer API. The Transformer API allows you to “fit” a preprocessing step using the training data the same way you’d fit a model…

…and then use the same transformation on future data sets!

Here’s what that process looks like:

1. Fit the transformer on the training set (saving the means and standard deviations)
2. Apply the transformer to the training set (scaling the training data)
3. Apply the transformer to the test set (using the same means and standard deviations)

This makes your final estimate of model performance more realistic, and it allows to insert your preprocessing steps into a cross-validation pipeline (more on this in Step 7).

Here’s how you do it:

Now, the scaler object has the saved means and standard deviations for each feature in the training set.

Let’s confirm that worked:

Note how we’re taking the scaler object and using it to transform the training set. Later, we can transform the test set using the exact same means and standard deviations used to transform the training set:

Notice how the scaled features in the test set are not perfectly centered at zero with unit variance! This is exactly what we’d expect, as we’re transforming the test set using the means from the training set, not from the test set itself.

In practice, when we set up the cross-validation pipeline, we won’t even need to manually fit the Transformer API. Instead, we’ll simply declare the class object, like so:

This is exactly what it looks like: a modeling pipeline that first transforms the data using StandardScaler() and then fits a model using a random forest regressor. Again, the random_state= parameter can be any number you choose. It’s simply setting the seed so that you get consistent results each time you run the code.

Step 6: Declare hyperparameters to tune.

Now it’s time to consider the hyperparameters that we’ll want to tune for our model.

WTF are hyperparameters?

There are two types of parameters we need to worry about: model parameters and hyperparameters. Models parameters can be learned directly from the data (i.e. regression coefficients), while hyperparameters cannot.

Hyperparameters express “higher-level” structural information about the model, and they are typically set before training the model.

Example: random forest hyperparameters.

As an example, let’s take our random forest for regression:

Within each decision tree, the computer can empirically decide where to create branches based on either mean-squared-error (MSE) or mean-absolute-error (MAE). Therefore, the actual branch locations are model parameters.

However, the algorithm does not know which of the two criteria, MSE or MAE, that it should use. The algorithm also cannot decide how many trees to include in the forest. These are examples of hyperparameters that the user must set.

We can list the tunable hyperparameters like so:

You can also find a list of all the parameters on the RandomForestRegressor documentation page. Just note that when it’s tuned through a pipeline, you’ll need to prepend  randomforestregressor__ before the parameter name, like in the code above.

Now, let’s declare the hyperparameters we want to tune through cross-validation.

As you can see, the format should be a Python dictionary (data structure for key-value pairs) where keys are the hyperparameter names and values are lists of settings to try. The options for parameter values can be found on the documentation page.

Step 7: Tune model using a cross-validation pipeline.

Now we’re almost ready to dive into fitting our models. But first, we need to spend some time talking about cross-validation.

This is one of the most important skills in all of machine learning because it helps you maximize model performance while reducing the chance of overfitting.

WTF is cross-validation (CV)?

Cross-validation is a process for reliably estimating the performance of a method for building a model by training and evaluating your model multiple times using the same method.

Practically, that “method” is simply a set of hyperparameters in this context.

These are the steps for CV:

1. Split your data into k equal parts, or “folds” (typically k=10).
2. Train your model on k-1 folds (e.g. the first 9 folds).
3. Evaluate it on the remaining “hold-out” fold (e.g. the 10th fold).
4. Perform steps (2) and (3) k times, each time holding out a different fold.
5. Aggregate the performance across all k folds. This is your performance metric.

Why is cross-validation important in machine learning?

Let’s say you want to train a random forest regressor. One of the hyperparameters you must tune is the maximum depth allowed for each decision tree in your forest.

How can you decide?

That’s where cross-validation comes in. Using only your training set, you can use CV to evaluate different hyperparameters and estimate their effectiveness.

This allows you to keep your test set “untainted” and save it for a true hold-out evaluation when you’re finally ready to select a model.

For example, you can use CV to tune a random forest model, a linear regression model, and a k-nearest neighbors model, using only the training set. Then, you still have the untainted test set to make your final selection between the model families!

So WTF is a cross-validation “pipeline?”

The best practice when performing CV is to include your data preprocessing steps inside the cross-validation loop. This prevents accidentally tainting your training folds with influential data from your test fold.

Here’s how the CV pipeline looks after including preprocessing steps:

1. Split your data into k equal parts, or “folds” (typically k=10).
2. Preprocess k-1 training folds.
3. Train your model on the same k-1 folds.
4. Preprocess the hold-out fold using the same transformations from step (2).
5. Evaluate your model on the same hold-out fold.
6. Perform steps (2) – (5) k times, each time holding out a different fold.
7. Aggregate the performance across all k folds. This is your performance metric.

Fortunately, Scikit-Learn makes it stupidly simple to set this up:

Yes, it’s really that easy. GridSearchCV essentially performs cross-validation across the entire “grid” (all possible permutations) of hyperparameters.

It takes in your model (in this case, we’re using a model pipeline), the hyperparameters you want to tune, and the number of folds to create.

Obviously, there’s a lot going on under the hood. We’ve included the pseudo-code above, and we’ll cover writing cross-validation from scratch in a separate guide.

Now, you can see the best set of parameters found using CV:

Interestingly, it looks like the default parameters win out for this data set.

*Tip: It turns out that in practice, random forests don’t actually require a lot of tuning. They tend to work pretty well out-of-the-box with a reasonable number of trees. Even so, these same steps can be used when building any type of supervised learning model.

Step 8: Refit on the entire training set.

After you’ve tuned your hyperparameters appropriately using cross-validation, you can generally get a small performance improvement by refitting the model on the entire training set.

Conveniently, GridSearchCV from sklearn will automatically refit the model with the best set of hyperparameters using the entire training set.

This functionality is ON by default, but you can confirm it:

Now, you can simply use the  clf object as your model when applying it to other sets of data. That’s what we’ll be doing in the next step.

Step 9: Evaluate model pipeline on test data.

Alright, we’re in the home stretch!

This step is really straightforward once you understand that the  clf object you used to tune the hyperparameters can also be used directly like a model object.

Here’s how to predict a new set of data:

Now we can use the metrics we imported earlier to evaluate our model performance.

Great, so now the question is… is this performance good enough?

Well, the rule of thumb is that your very first model probably won’t be the best possible model. However, we recommend a combination of three strategies to decide if you’re satisfied with your model performance.

1. Start with the goal of the model. If the model is tied to a business problem, have you successfully solved the problem?
2. Look in academic literature to get a sense of the current performance benchmarks for specific types of data.
3. Try to find low-hanging fruit in terms of ways to improve your model.

There are various ways to improve a model. We’ll have more guides that go into detail about how to improve model performance, but here are a few quick things to try:

1. Try other regression model families (e.g. regularized regression, boosted trees, etc.).
2. Collect more data if it’s cheap to do so.
3. Engineer smarter features after spending more time on exploratory analysis.
4. Speak to a domain expert to get more context (…this is a good excuse to go wine tasting!).

As a final note, when you try other families of models, we recommend using the same training and test set as you used to fit the random forest model. That’s the best way to get a true apples-to-apples comparison between your models.

Step 10: Save model for future use.

Great job completing this tutorial!

You’ve done the hard part, and deserve another glass of wine. Maybe this time you can use your shiny new predictive model to select the bottle.

But before you go, let’s save your hard work so you can use the model in the future. It’s really easy to do so:

And that’s it. When you want to load the model again, simply use this function:

Congratulations, you’ve reached the end of this tutorial!

We’ve just completed a whirlwind tour of Scikit-Learn’s core functionality, but we’ve only really scratched the surface. Hopefully you’ve gained some guideposts to further explore all that sklearn has to offer.

For continued learning, we recommend studying other examples in sklearn.

The complete code, from start to finish.

Here’s all the code in one place, in a single script.