4. Evaluation Metrics for Classification¶

Last week we trained a model for churn. How do we know if it’s good?

The fourth week of Machine Learning Zoomcamp is about different metrics to evaluate a binary classifier. These measures include accuracy, confusion table, precision, recall, ROC curves(TPR, FRP, random model, and ideal model), AUROC, and cross-validation.

4.1 Evaluation metrics: session overview¶

• Dataset: https://www.kaggle.com/blastchar/telco-customer-churn
• https://raw.githubusercontent.com/alexeygrigorev/mlbookcamp-code/master/chapter-03-churn-prediction/WA_Fn-UseC_-Telco-Customer-Churn.csv

Metric – function that compares the predictions with the actual values and outputs a single number that tells how good the predictions are

4.2 Accuracy and dummy model¶

Accurcy measures the fraction of correct predictions. Specifically, it is the number of correct predictions divided by the total number of predictions.

We can change the decision threshold, it should not be always 0.5. But, in this particular problem, the best decision cutoff, associated with the hightest accuracy (80%), was indeed 0.5.

Note that if we build a dummy model in which the decision cutoff is 1, so the algorithm predicts that no clients will churn, the accuracy would be 73%. Thus, we can see that the improvement of the original model with respect to the dummy model is not as high as we would expect.

Therefore, in this problem accuracy can not tell us how good is the model because the dataset is unbalanced, which means that there are more instances from one category than the other. This is also known as class imbalance.

Classes and methods:

• np.linspace(x,y,z) – returns a numpy array starting at x until y with a z step
• Counter(x) – collection class that counts the number of instances that satisfy the x condition
• accuracy_score(x, y) – sklearn.metrics class for calculating the accuracy of a model, given a predicted x dataset and a target y dataset.

So we have the accuracy of our base model using 0.5 as the cutoff for churn, no churn. Now we will vary the 0.5 to different numbers to see if the accuracy of our model is better or not.

The np.linspace() method can be used to create an array. In this case we want the values in the array to be from 0 to 1 and we want 21 of them. That will start at 0 and increment by .05 each step.

We can see that 0.5 is the best one. We used our own function for calculating accuracy (y_val == churn_decision).mean(). sklearn has a function for accuracy, accuracy_score. Let’s implement that.

Let’s add that into our previous function to calculate accuracy with the thresholds array.

At the threshold of 1.0 we can see our prediction rate is 72.6%. That must mean that our churn rate in the validation set is 72.6%. Let’s verify that using the Counter method, which simply counts the number of True or False.

As we can see the number of False records is at 1409, which is the size of our array, so none of the values are greater than 1. Below we can simply do a small calculation and see that it is simply a calculation of the churn rate of our dataset.

So the actual non churn rate is at 72.6% for the validation dataset and our model is only 7.4% better. So accuracy isn’t the best choice for scoring our model. This is because we have what is called class imbalance. We have 72.6% of customers that don’t churn compared to 27.4% of customers that do churn.

4.3 Confusion table¶

Confusion table is a way to measure different types of errors and correct decisions that binary classifiers can made. Considering this information, it is possible evaluate the quality of the model by different strategies.

If we predict the probability of churning from a customer, we have the following scenarios:

• No churn – Negative class
  • Customer did not churn – True Negative (TN)
  • Customer churned – False Negative (FN)
• Churn – Positive class
  • Customer churned – True Positive (TP)
  • Customer did not churn – False Positive (FP)

The confusion table help us to summarize the measures explained above in a tabular format, as is shown below:

The accuracy corresponds to the sum of TN and TP divided by the total of observations.

In effect, what we are setting up is a logic AND argument. If both are True then the output is True, if both are False then the output is True, if either is different then we get an output that is False

We have 210 True Positives and 922 True Negatives

We have 101 False Positives and 176 False Negatives

What all of this is telling us is that with our current model we would be sending out 101 discount emails to people who not at risk of churning and we would be missing out by not sending out 176 discount emails to people who were going to churn. In the first case we are losing money because we gave discounts to people who were not at risk of churning and in the second case we are missing out on future money by not attempting to retain customers that are at high risk for churning.

4.4 Precision and Recall¶

Precision tell us the fraction of positive predictions that are correct. It takes into account only the positive class (TP and FP – second column of the confusion matrix), as is stated in the following formula:

TP / (TP + FP)

Recall measures the fraction of correctly identified postive instances. It considers parts of the postive and negative classes (TP and FN – second row of confusion table). The formula of this metric is presented below:

TP / (TP + FN)

In this problem, the precision and recall values were 67% and 54% respectively. So, these measures reflect some errors of our model that accuracy did not notice due to the class imbalance.

Precision¶

In the precision score we are only interested in scores that equal churn. In this case we would be interested in the TP and FP.

Recall¶

4.5 ROC Curves¶

ROC stands for Receiver Operating Characteristic, and this idea was applied during the Second World War for evaluating the strenght of radio detectors. This measure considers False Positive Rate (FPR) and True Postive Rate (TPR), which are derived from the values of the confusion matrix.

FPR is the fraction of false positives (FP) divided by the total number of negatives (FP and TN – the first row of confusion matrix), and we want to minimize it. The formula of FPR is the following:

In the other hand, TPR or Recall is the fraction of true positives (TP) divided by the total number of positives (FN and TP – second row of confusion table), and we want to maximize this metric. The formula of this measure is presented below:

ROC curves consider Recall and FPR under all the possible thresholds. If the threshold is 0 or 1, the TPR and Recall scores are the opposite of the threshold (1 and 0 respectively), but they have different meanings, as we explained before.

We need to compare the ROC curves against a point of reference to evaluate its performance, so the corresponding curves of random and ideal models are required. It is possible to plot the ROC curves with FPR and Recall scores vs thresholds, or FPR vs Recall.

Classes and methods:

• np.repeat([x,y], [z,w]) – returns a numpy array with a z number of x values, and a w number of y values.
• roc_curve(x, y) – sklearn.metrics class for calculating the false positive rates, true positive rates, and thresholds, given a target x dataset and a predicted y dataset.

TPR and FPR¶

TPR is the same equation as recall

Random model¶

Ideal model¶

The np.repeat function will create an array with 0’s the number of times of num_neg and 1’s the number of times of num_pos. This creates the order in the second set in the image.

Putting it all together¶

4.6 ROC AUC¶

The Area under the ROC curves can tell us how good is our model with a single value. The AUROC of a random model is 0.5, while for an ideal one is 1.

In ther words, AUC can be interpreted as the probability that a randomly selected positive example has a greater score than a randomly selected negative example.

Classes and methods:

• auc(x, y) – sklearn.metrics class for calculating area under the curve of the x and y datasets. For ROC curves x would be false positive rate, and y true positive rate.
• roc_auc_score(x, y) – sklearn.metrics class for calculating area under the ROC curves of the x false positive rate and y true positive rate datasets.

Below is an example of how roc_auc_score is actually calculated

We can also do this with Numpy

4.7 Cross-Validation¶

Extra resources In the lesson we talked about iterators and generators in Python. You can read more about them here:

• https://anandology.com/python-practice-book/iterators.html
• https://www.google.com/search?q=python+iterators+and+generators

Notes

Cross-validation refers to evaluating the same model on different subsets of a dataset, getting the average prediction, and spread within predictions. This method is applied in the parameter tuning step, which is the process of selecting the best parameter.

In this algorithm, the full training dataset is divided into k partitions, we train the model in k-1 partiions of this dataset and evaluate it on the remaining subset. Then, we end up evaluating the model in all the k folds, and we calculate the average evaluation metric for all the folds.

In general, if the dataset is large, we should use the hold-out validation dataset strategy. In the other hand, if the dataset is small or we want to know the standard deviation of the model across different folds, we can use the cross-validation approach.

Libraries, classes and methods:

• Kfold(k, s, x) – sklearn.model_selection class for calculating the cross validation with k folds, s boolean attribute for shuffle decision, and an x random state
• Kfold.split(x) – sklearn.Kfold method for splitting the x dataset with the attributes established in the Kfold’s object construction.
• for i in tqdm() – library for showing the progress of each i iteration in a for loop.

tqdm is a package that allows us to see progress bars during iteration functions.

Above we see the scores and they are all very similar, 0.001 is the lowest. All the rest are very close and since C=1.0 is the default we will just use that. Since all the scores were very similar we can run the model on the test data set.

4.8 Summary¶

General definitions:

• Metric: A single number that describes the performance of a model
• Accuracy: Fraction of correct answers; sometimes misleading
• Precision and recall are less misleading when we have class inbalance
• ROC Curve: A way to evaluate the performance at all thresholds; okay to use with imbalance
• K-Fold CV: More reliable estimate for performance (mean + std)

In brief, this weeks was about different metrics to evaluate a binary classifier. These measures included accuracy, confusion table, precision, recall, ROC curves(TPR, FRP, random model, and ideal model), and AUROC. Also, we talked about a different way to estimate the performance of the model and make the parameter tuning with cross-validation.

4.9 Explore more¶

• Check the precision and recall of the dummy classifier that always predict “FALSE”
• F1 score = 2 P R / (P + R)
• Evaluate precision and recall at different thresholds, plot P vs R – this way you’ll get the precision/recall curve (similar to ROC curve)
• Area under the PR curve is also a useful metric

Other projects

• Calculate the metrics for the suggested datasets from the previous week