ROC/AUROC Curve Explained
The goal of this blog post is to explain to those who aren’t familiar with it, the ROC curve, what it tells us, and how we can use it to compare the strength of our models.
What can the ROC/AUROC curves tell us?
- First and foremost, the ROC and AUC curves go above and beyond the traditional measures of classification metrics. Usually, many will opt to use ‘accuracy’ of a model, when it might not capture the entire picture of the classifier you’re trying to build. Simply put, the ROC curve can tell us how well our model is interpreting the 1 class as 1, and the 0 class as 0.
For example:
If we have a dataset that contains the following binary values for heart disease in men:
Heart Disease: [0,0,0,0,1,1]
Weight: [150,150,190,210,240,250]
We can see that if we were to set the threshold for predicting heart disease at 200lbs, we would misclassify someone who is healthy, incorrectly as someone with heart disease. Based on our data, if we selected 225lbs as the threshold for the classifier, we would correctly predict 100% of the values and thus it makes much more sense to use as the threshold by which to classify.
Additionally, these curves come along with a few different rates/metrics that we can look at.
TP = True Positives
FN = False Negative
FP = False Positive
FN = False Negative
The true positive rate shows how well the model does at predicting the positive class when the actual value is positive.
- True Positive Rate = TP / (TP + FN)
This rate has multiple names and can be referred to as ‘Sensitivity’ or the ‘false alarm rate’. This is because it can classify predicted positive classes as positive when the actual outcome is negative.
2. False Positive Rate = FP / (FP + TN)
This rate is the proportion of the data that has a known negative condition for which the predicted condition is positive. It can be referred to as the ‘fall-out’, and is seen along the X axis of the ROC curve. (We have to sometimes put up with this being a high rate because it can be especially important in certain areas of life where incorrectly identifying something airs on the side of safety, such as identify disease in patients, or judging whether or not someone is pregnant.)
The ROC curve can be helpful to us in a number of ways.
We can use ROC curves to decide on a threshold value (as mentioned previously). The threshold depends heavily on the use of the classifier in question. Usually the cost-reward ratio will be weighed to help determine this.
The AUC just stands for ‘area under the curve’ and is represented as a value between 0 and 1. (An integral is used to solve for this.) We want to maximize our AUC, because it will give us the highest TPR and lowest FPR (which means our model is running well given the threshold).
Wrapping up, a big part of evaluating models, is measuring them against each other to see how they stack up. If you don’t understand the metric you’re using, or you pick and choose the wrong one to evaluate your model, it could potentially give you less than ideal results. Keep this information in mind when you’re deciding how to evaluate your next binary classification problem.