PYTHON / MACHINE LEARNING WITH PYTHON
Evaluating models: the metrics that matter
Read a confusion matrix and choose between accuracy, precision, recall, F1, MAE, RMSE and R2 based on which errors actually cost you something.
What you will learn
- Derive precision, recall and F1 from TP, FP, FN by hand and with sklearn
- Show why accuracy is useless on a 5%-positive dataset
- Move a decision threshold and predict which metric rises and which falls
- Pick MAE, RMSE or R2 for a regression task and justify the choice
Understanding Evaluating models: the metrics that matter
Every classification metric is a different summary of the same four numbers: true positives, false positives, false negatives and true negatives. Accuracy adds TP and TN and divides by the total, which silently assumes a false positive costs exactly as much as a false negative. When 5 of 100 patients are sick, predicting 'healthy' for everyone scores 0.95 accuracy while finding nobody, so the number is high precisely because the errors it ignores are the only ones that matter.
Precision and recall split accuracy along the axis you care about. Precision is TP/(TP+FP): of the cases you flagged, how many deserved it, which is the question a user asks when a false alarm wastes their time. Recall is TP/(TP+FN): of the cases that existed, how many you found, which is the question asked when a miss is expensive. F1 is their harmonic mean rather than the arithmetic mean, so a model with precision 1.0 and recall 0.0 gets F1 0.0 instead of 0.5.
Most classifiers do not output labels, they output scores, and a label appears only after you compare that score to a threshold. Precision and recall therefore describe an operating point, not the model, and lowering the threshold always pushes recall up and precision down as more borderline cases get flagged. For regression the same logic applies to the loss shape: MAE averages absolute errors so each mistake counts once, RMSE squares them first so one large miss dominates, and R2 compares your errors to the errors of always predicting the mean, which is why it goes negative when you do worse than that baseline.
Choose the metric before you look at the scores, otherwise you will pick whichever one flatters the model you already trained.
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, confusion_matrix)
# 100 patients, only 5 actually have the disease
y_true = [1] * 5 + [0] * 95
# Model A: predicts "healthy" for everyone
always_negative = [0] * 100
# Model B: catches 4 of the 5 sick patients, raises 6 false alarms
model_b = [1, 1, 1, 1, 0] + [1] * 6 + [0] * 89
for name, y_pred in [("always-negative", always_negative), ("model B", model_b)]:
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
print(name)
print(" TP={} FP={} FN={} TN={}".format(tp, fp, fn, tn))
print(" accuracy {:.3f}".format(accuracy_score(y_true, y_pred)))
print(" precision {:.3f}".format(precision_score(y_true, y_pred, zero_division=0)))
print(" recall {:.3f}".format(recall_score(y_true, y_pred, zero_division=0)))
print(" f1 {:.3f}".format(f1_score(y_true, y_pred, zero_division=0)))A metric is a choice about which kind of error you are willing to make, so pick it from the cost of the mistakes rather than from the confusion matrix you got.
Worked examples
The threshold owns precision and recall
The same set of predicted probabilities gives three different precision/recall pairs depending only on where you cut.
from sklearn.metrics import precision_score, recall_score
y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]
y_proba = [0.95, 0.80, 0.55, 0.30, 0.60, 0.45, 0.20, 0.15, 0.10, 0.05]
for t in (0.5, 0.35, 0.25):
y_pred = [1 if p >= t else 0 for p in y_proba]
print("threshold {:.2f} precision {:.2f} recall {:.2f}".format(
t,
precision_score(y_true, y_pred),
recall_score(y_true, y_pred)))Example explained
Line 1y_proba never changes, so the model is fixed and only the cut point moves.
Line 2At 0.25 the true positive with score 0.30 finally gets flagged, so recall reaches 1.00.
Line 3At 0.35 two negatives (0.60 and 0.45) are already flagged, dragging precision to 0.60 while recall stands still.
Line 4Reporting a single precision number without saying which threshold produced it describes nothing reproducible.
MAE, RMSE and R2 on one outlier
One badly predicted point barely moves MAE but multiplies RMSE and sends R2 negative.
import math
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
y_true = [10, 12, 14, 16, 18]
runs = {"tight fit": [11, 12, 13, 16, 19],
"one big miss": [11, 12, 13, 16, 34]}
for name, y_pred in runs.items():
mae = mean_absolute_error(y_true, y_pred)
rmse = math.sqrt(mean_squared_error(y_true, y_pred))
print("{:13} MAE {:.2f} RMSE {:.2f} R2 {:.3f}".format(
name, mae, rmse, r2_score(y_true, y_pred)))Example explained
Line 1The two runs differ in one prediction only: 19 becomes 34, an error of 16 instead of 1.
Line 2MAE goes 0.60 to 3.60 because that single error is averaged in linearly, one point out of five.
Line 3RMSE goes 0.77 to 7.18 because 16 is squared to 256 before averaging, so the outlier dominates the sum.
Line 4R2 -5.450 means the model's squared error is over six times that of simply always predicting the mean, 14.
Important notes
precision_score, recall_score and f1_score default to average='binary'; on three or more classes you must pass average='macro' or 'weighted', and those two disagree badly when class sizes differ.
zero_division=0 silences the warning when a model predicts no positives at all, but a precision of 0.0 there means undefined, not measured-and-bad.
Common mistakes
Quoting accuracy on imbalanced data: a fraud model on 0.2% fraud reports 0.998 accuracy while flagging nothing, and the team ships a detector with recall 0.
Passing hard 0/1 labels to roc_auc_score instead of predict_proba scores: the AUC collapses toward the single-threshold value and looks far worse than the model actually is.
Sweeping thresholds until F1 peaks on the test set, then reporting that F1: the number is now a fitted parameter, and real traffic scores lower.
Try it yourself
Change, predict, then run
Build y_true with 3 positives and 20 negatives, score an all-zeros prediction with accuracy_score and recall_score, then flip two predictions to 1 (one correct, one wrong) and print precision and recall again. Report which of the four numbers moved and which stayed nearly unchanged.
Open the Python workspaceCheck your understanding
A spam filter reports precision 0.99 and recall 0.40 at its default 0.5 threshold, and users complain that too much spam reaches the inbox. What is the most sensible first move?
- Lower the decision threshold, accepting more false positives in exchange for higher recall
- Retrain with more features, since precision 0.99 shows the model lacks capacity
- Report F1 instead of precision and recall so the summary number looks balanced
- Raise the threshold so the filter is more confident before flagging a message as spam
Show answer
Spam reaching the inbox is a false negative, which is exactly what recall measures, and lowering the threshold flags more borderline messages so recall rises while precision falls. Raising the threshold does the opposite and makes the complaint worse; changing which metric you report changes no prediction at all.