2. Machine Learning for Regression¶

• 2.1 Car price prediction project
• 2.2 Data preparation
• 2.3 Exploratory data analysis
• 2.4 Setting up the validation framework
• 2.5 Linear regression
• 2.6 Linear regression: vector form
• 2.7 Training linear regression: Normal equation
• 2.8 Baseline model for car price prediction project
• 2.9 Root mean squared error
• 2.10 Using RMSE on validation data
• 2.11 Feature engineering
• 2.12 Categorical variables
• 2.13 Regularization
• 2.14 Tuning the model
• 2.15 Using the model
• 2.16 Car price prediction project summary

2.1 Car price prediction project¶

Project plan:

• Prepare data and Exploratory data analysis (EDA)
• Use linear regression for predicting price
• Understanding the internals of linear regression
• Evaluating the model with RMSE
• Feature engineering
• Regularization
• Using the model

First we import the libraries for manipulating data. Pandas is a python package used to analyze and manipulate data. NumPy is a python package used to work with arrays.

2.2 Data preparation¶

Pandas attributes and methods:

• pd.read.csv() – read csv files
• df.head() – take a look of the dataframe
• df.columns – retrieve colum names of a dataframe
• df.columns.str.lower() – lowercase all the letters
• df.columns.str.replace(' ', '_') – replace the space separator
• df.dtypes – retrieve data types of all features
• df.index – retrieve indices of a dataframe

Using wget to download the data needed for this regression

Load the file using pandas read read_csvfunction

df.columns is part of the Pandas library that will return the column names of the dataframe.

We want to clean up our column names to remove caps and replace spaces with underscores and the operations can be chained together and reassigned to the columns variable. The str.lower() is a method for altering the selected string to all lower case. The str.replace() method can be used to replace specific characters in the string, in this case we want to replace spaces in column names with an underscore so that we can use dot methods. Dot methods will not work on strings with spaces.

Now when we view the dataframe we can see that the column names no longer contain spaces and everything is lower case.

The df.dytpes shows us what type of data is contained in each column. We are particularlly interested in the “object” columns as they contain data in a string format.

We can see in the dataframe that the data is also inconsistant in its formatting. Lets clean that up and we start by identifying which columns contain a string format, which is equated by the data type object

Now we want to select only those columns (or ‘features’) that are of the data type ‘object’

Now that we know which features are objects we don’t need the return showing us the data types and we can ‘unselect’ those by using .index to just retrieve the index

Lets transform those “object” containing column names to a python list and and assign the list to the variable named “strings”.

Now we can create a short function that will iterate through each column in the variable list named “strings” and apply the .lower abd .replace methods to data contained within the column. After we call the df again we can see that all the ‘object’ data with in the dataframe has been transformed to remove spaces and remove all capitlization.

2.3 Exploratory data analysis¶

Pandas attributes and methods:

• df[col].unique() – returns a list of unique values in the series
• df[col].nunique() – returns the number of unique values in the series
• df.isnull().sum() – retunrs the number of null values in the dataframe

Matplotlib and seaborn methods:

• %matplotlib inline – assure that plots are displayed in jupyter notebook’s cells
• sns.histplot() – show the histogram of a series

Numpy methods:

• np.log1p() – applies log transformation to a variable and adds one to each result

Long-tail distributions usually confuse the ML models, so the recommendation is to transform the target variable distribution to a normal one whenever possible.

We want to learn more about the data that we are using. We can use some simple functions to more fully understand what the data contains.

The above doesn’t give us much detail, for instance, under ‘make’ it shows all bmw’s but we can use the unique and nunique functions to get a little better understanding of what our data contains. This function will show us the first five unique items in each column and count the number of unique items in each column. For instance we can see there are 48 different manufacturers of cars in the dataset.

Let us see what the distribution of prices are, we can use visualization to do this. We will import Matplotlib and Seaborn. Seaborn is a package built on top of the Matplotlib library that allows enhanced visualizations on data. The %matplotlib inline statement is notifying Jupyter Notebook that we will be doing some plotting.

It isn’t a very good plot right off the bat, we can increase it’s readability.

That is a very long tail, with most of the car prices grouped near the zeros and some cars valued up around 2 million. We will filter out some of the high price outliers to, perhaps, get a better representation of the data.

Our data has that very long tail, which will confuse our model. We will have to do something to change this. Changing the prices to a logarithm scale, will group our prices closer. A standard practice in converting a log scale is adding 1 to each data point. This ensures that there are no errors in the event that one of the points is a zero. For now we are doing this just to the plot the data and see that it is more of a normal distribution. Later we will apply this log transformation to the “msrp” column of our dataset.

Now we have our prices close together, our graph should look better.

We have eliminated that long tail off to the right and now our data looks more like a normal distribution.

Missing values¶

2.4 Setting up the validation framework¶

In general, the dataset is split into three parts: training, validation, and test. For each partition, we need to obtain feature matrices (X) and y vectors of targets. First, the size of partitions is calculated, records are shuffled to guarantee that values of the three partitions contain non-sequential records of the dataset, and the partitions are created with the shuffled indices.

Pandas attributes and methods:

• df.iloc[] – returns subsets of records of a dataframe, being selected by numerical indices
• df.reset_index() – restate the orginal indices
• del df[col] – eliminates target variable

Numpy methods:

• np.arange() – returns an array of numbers
• np.random.shuffle() – returns a shuffled array
• np.random.seed() – set a seed

Let’s draw it

We have 11,914 records that need to be separated into training, validation and testing sets. Lets calculate how many records will be in each set using len() and some math.

So our training set will contain 7,150 records, and the validation and test sets will contain 2382 each.

Now we could just grab our records using .iloc and the new variables we just set up.

The problem with breaking up our records like this will be that it isn’t a cross-section of records. For example, df_train will contain all of the bmw vehicles. So we need to do something to fhuffle the records around.

We use the np.arange method to sort the dataframe and then assign it to the variable “idx”.

Then we can use .random.shuffle() to shuffle everything around so that all the bmw’s are not in the train test set.

Since I am doing this as part of the ml-zoomcamp class we want us to have the same shuffle for our data that our instructor has. We can do this by using the np.random.seed() method. and then we will do the suffle again.

Now that we have a randomly shuffled dataframe we want to reset the index so that is in numerical order.

Before we start the regression problem we need to apply the log function to the “msrp” column so that the data doesn’t have that long tail.

We will use the np.log1p function to perform this on the data.

The data above is a series, we will convert that to a pandas array and then assign it to the “y_train” variable and then we will do that to the other sets as well.

None of our df sets, i.e. df_test, need the msrp column because that is what we are trying to predict. We will delete that column from our df sets.

2.5 Linear regression¶

Model for solving regression tasks, in which the objective is to adjust a line for the data and make predictions on new values. The input of this model is the feature matrix and a y vector of predictions is obtained, trying to be as close as possible to the actual y values. The LR formula is the sum of the bias term (WO), which refers to the predictions if there is no information, and each of the feature values times their corresponding weights. We need to assure that the result is shown on the untransformed scale.

So our model is g, df_train, df_val and df_test are matricour feature matrices and our targets are the variables we created with y_train, y_val and y_test

We can look at different records in our dataset, below we are looking at the row indexed at 10 using the .loc[] method.

We are going to choose engine_hp, city_mpg and popularity.

Now we will right a short function to predict a price based on the feature set we chose.

Each feature in xi will be assigned a weight to either increase or decrease the efficacy on the price of a vehicle.

With Python we start at 0 so it would be from 0 to 2 for our 3 features.

So our prediction was 550 for the price of the vehicle, in log + 1. It isn’t correct because w0 and w aren’t the correct starting points, that is where machine learning comes in but lets keep working at this solution. Keep in mind the returns we are getting are in a log price that have to back-transform the log. Also, because we did np.log1p() which added a 1, we can use np.expm1 to back-transform and subtract 1.

Back-transform the log.

The value above is our first attempt and predicting the price of the Rolls-Royce @ 222,347

2.6 Linear regression: vector form¶

The formula of LR can be synthesized with the dot product between features and weights. The feature vector includes the bias term with an x value of one. When all the records are included, the LR can be calculated with the dot product between feature matrix and vector of weights, obtaining the y vector of predictions.

Now we will do another prediction on another record (making some numbers up just for the example)

2.7 Training linear regression: Normal equation¶

Obtaining predictions as close as possible to y target values requires the calculation of weights from the general LR equation. The feature matrix does not have an inverse because it is not square, so it is required to obtain an approximate solution, which can be obtained using the Gram matrix (multiplication of feature matrix and its transpose). The vector of weights or coefficients obtained with this formula is the closest possible solution to the LR system.

We need to add our bias terms into the array, which will be 1’s. We will take the shape of the array above and assign the row count to the np.ones method to create a vector of 1’s to match the matrix and then we will append that vector to the matrix

We need to create a y vector.

We now create our inverse matrix, which actuall creates an identity matrix.

We can use the dot product of XTX and XTX_inv to see if we have created an identity matrix, and we have.

2.8 Baseline model for car price prediction project¶

The LR model obtained in the previous section was used with the dataset of car price prediction. For this model, only the numerical variables were considered. The training data was pre-processed, replacing the NaN values with 0, in such a way that these values were omitted by the model. Then, the model was trained and it allowed to make predictions on new data. Finally, distributions of y target variable and predictions were compared by plotting their histograms.

We are going to build a model using the features engine_hp, engine_cylinders, highway_mgp, city_mpg, and popularity.

We assign the columns we want to create a model on and assign it to the variable ‘base’.

Next, we need to extract the values from our columns to use in our model and we assign those values to the variable ‘X_train’.

We have already created our ‘y’ with the MSRP column that we removed from ‘df_train’ earlier.

With the ‘X_train’ and ‘y_train’ variables we can insert these into the train_linear_regression function we created earlier.

As we can see above we have some ‘nan’ values, which we can’t have. We noticed earlier that we had some null values in the dataset, we have fill those in with some values.

As we can see above we have 40 empty entries for ‘engine_hp’ and 14 for ‘engine_cylinders’. There are various ways to fill in these values. We can use the median or mean values of the columns or fill them with a 0.

Now that we have filled in the nan values we can see that our function now works properly.

Let’s assign the model to our bias term and w.

Now we can do the matrix to matrix multiplication to get our predictions.

Let’s assign our predictions to the variable ‘y_pred’.

Let’s plot our ‘y_preds’ to see how they comapre to ‘y_train’, which are the actual MSRP’s from our data, which are also known as the target variables. We will can assign different colors for each, the ‘alpha’ argument changes the transparency so that we can see both and the ‘bins’ argument breaks the data into 50 segments.

We can see in the plot that our predicted prices are lower than actual prices and the mean of the prices peak is also low. Even though our model isn’t perfect we can use it to evaluate other models against it.

2.9 Root mean squared error¶

The RMSE is a measure of the error associated with a model for regression tasks. The video explained the RMSE formula in detail and implemented it in Python. This will be how we quanitfy the accuracy of the previous built model.

2.10 Using RMSE on validation data¶

Calculation of the RMSE on validation partition of the dataset of car price prediction. In this way, we have a metric to evaluate the model’s performance. The code below is what we needed to train our model on the X_train dataset, now we will set it up to train on the validation set that we created earlier in this notebook.

2.11 Feature engineering¶

The feature age of the car was included in the dataset, obtained with the subtraction of the maximum year of cars and each of the years of cars. This new feature improved the model performance, measured with the RMSE and comparing the distributions of y target variable and predictions.

The age of a car is a good prediction of price. The older the car the lower the price. We can calculate the age of the car by subtracing the year of the car from the max() year in the dataset.

We can use the function we created earlier and modify it to add the column ‘age’ as one of the features in our dataset.

After we run our predictions again we can see the RMSE decreased from 0.7554192603920132 down to 0.5172055461058329. The smaller the RMSE number is the closer our predictions are to the known prices.

Visually we can see the graph is closer to the known prices.

We are getting closer, there looks to be an issue with some of the lower priced cars.

2.12 Categorical variables¶

Categorical variables are typically strings, and pandas identify them as object types. These variables need to be converted to a numerical form because the ML models can interpret only numerical features. It is possible to incorporate certain categories from a feature, not necessarily all of them. This transformation from categorical to numerical variables is known as One-Hot encoding.

Above we can see that our ‘prepare’ function added our categorical columns for number of doors.

There is a very slight improvement using doors as a feature. Let’s look at adding the ‘Make’ as a feature.

As we can see, that made another slight improvement to our model by increasing it’s accuracy a little more.

2.13 Regularization¶

If the feature matrix has duplicated columns, it does not have an inverse matrix. But, sometimes this error could be passed if certain values are slightly different between duplicated columns.

So, if we apply the normal equation with this feature matrix, the values associated with duplicated columns are very large, which decreases the model performance. To solve this issue, one alternative is adding a small number to the diagonal of the feature matrix, which corresponds to regularization.

This technique works because the addition of small values to the diagonal makes it less likely to have duplicated columns. The regularization value is a parameter of the model. After applying regularization the model performance improved.

2.14 Tuning the model¶

The tuning consisted of finding the best regularization value, using the validation partition of the dataset. After obtaining the best regularization value, the model was trained with this regularization parameter.

2.15 Using the model¶

After finding the best model and its parameters, it was trained with training and validation partitions and the final evaluation was calculated on the test partition. Finally, the final model was used to predict the price of new cars.

Now that we validated our model with the validation set, it can be added back in with the training data set so that we have one large training set, which comprises 80% of the original dataset.

Above we can see that there are 9532 rows but the index is still set from the validation dataset so we will reset the index.