PYTHON / CAPSTONE PROJECTS
Project: a classifier with an honest evaluation
Evaluate a classifier honestly: hold rows out, compare against a majority-class baseline, and report confusion-matrix metrics instead of bare accuracy.
What you will learn
- Print a majority-class baseline before you trust any accuracy figure
- Derive precision, recall and F1 from the four confusion-matrix counts by hand
- Read a train-vs-test accuracy gap as direct evidence of memorisation
- Stratify the split and pick the decision threshold on validation, never on test
Understanding Project: a classifier with an honest evaluation
The held-out set below has 2 positives in 20 rows, so a 'classifier' that returns 0 for every input is right 90% of the time. That is why the script prints the majority-class baseline first: an accuracy number measures the class balance at least as much as it measures the model. Treat every score as a comparison against the cheapest possible rule, and a lot of impressive figures collapse on contact.
Every class-aware metric falls out of four counts: true positives, false positives, false negatives, true negatives. The threshold rule scores 0.80 accuracy, ten points below the baseline, yet it is the only one of the two that catches a churner at all: recall 1.00 versus 0.00. Precision 0.33 states the price, four false alarms for two real hits, and F1 is the harmonic mean of those two, which is why it lands at 0.50 near the weaker number rather than averaging up to 0.67.
Honesty lives in the split, not in the metric. Anything decided after looking at test rows leaks into the reported score: the threshold, the feature list, the scaler's mean, even 'let me try one more model'. Fit on train, tune the threshold on a validation slice, touch test exactly once, and split each class separately so both parts carry the same positive rate. With 20 rows one flipped prediction moves accuracy by 5 points, so publish the counts next to the ratios.
# Hold-out rows the model never saw: (model score, true label)
# label 1 = churned. Only 2 of the 20 rows are positive.
held_out = [
(0.10, 0), (0.15, 0), (0.20, 0), (0.22, 0), (0.25, 0),
(0.30, 0), (0.31, 0), (0.35, 0), (0.40, 0), (0.44, 0),
(0.48, 0), (0.52, 0), (0.55, 0), (0.58, 0), (0.61, 0),
(0.65, 0), (0.70, 1), (0.72, 0), (0.88, 1), (0.91, 0),
]
y_true = [y for _, y in held_out]
scores = [s for s, _ in held_out]
def report(name, truth, pred):
tp = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 1)
fp = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 0)
tn = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 0)
acc = (tp + tn) / len(truth)
prec = tp / (tp + fp) if tp + fp else 0.0
rec = tp / (tp + fn) if tp + fn else 0.0
f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
print(f"{name:<18} acc={acc:.2f} prec={prec:.2f} rec={rec:.2f} f1={f1:.2f}"
f" tp={tp} fp={fp} fn={fn} tn={tn}")
print(f"positives in hold-out: {sum(y_true)}/{len(y_true)}")
report("majority baseline", y_true, [0] * len(y_true))
report("model @ 0.60", y_true, [1 if s >= 0.60 else 0 for s in scores])A classifier's score only means something next to a trivial baseline and next to metrics that separate the two kinds of error, measured on rows no decision was based on.
Worked examples
The train/test gap exposes memorisation
A model that stores its training rows in a dict scores perfectly on them and no better than a coin flip on unseen ones.
rows = [(1, 0), (2, 0), (3, 1), (4, 0), (5, 1), (6, 1), (7, 0), (8, 1)]
train, test = rows[:6], rows[6:]
lookup = {x: y for x, y in train}
def predict(x):
return lookup.get(x, 0) # unseen id -> majority class
for name, part in (("train", train), ("test", test)):
hits = sum(predict(x) == y for x, y in part)
print(f"{name}: {hits}/{len(part)} correct -> acc={hits / len(part):.2f}")Example explained
Line 1lookup stores every training pair, so training accuracy is 1.00 by construction, not by skill.
Line 2lookup.get(x, 0) falls back to the majority class for ids 7 and 8, which the model has never seen.
Line 3The gap between 1.00 and 0.50 is the reason a hold-out set exists: the first number measures memory, the second generalisation.
Line 4Any evaluation run on rows the model was fitted on produces the left-hand number and calls it performance.
Precision and recall move in opposite directions
Sweeping the decision threshold over one fixed set of scores shows there is no single 'accuracy of the model'.
scored = [(0.91, 0), (0.88, 1), (0.72, 0), (0.70, 1), (0.65, 0), (0.61, 0),
(0.58, 0), (0.52, 0), (0.44, 0), (0.40, 0), (0.31, 0), (0.20, 0)]
positives = sum(y for _, y in scored)
for thr in (0.40, 0.60, 0.80):
flagged = [(s, y) for s, y in scored if s >= thr]
tp = sum(y for _, y in flagged)
prec = tp / len(flagged)
rec = tp / positives
print(f"thr={thr:.2f} flagged={len(flagged):2d} prec={prec:.2f} rec={rec:.2f}")Example explained
Line 1Raising thr can only shrink the flagged list, so precision rises or stays flat while recall can only fall.
Line 2At 0.60 the model still catches both positives but sends six rows for review instead of ten: same recall, half the work.
Line 3At 0.80 it misses one positive, so recall 0.50 is exactly what precision 0.50 cost.
Line 4The threshold is a cost decision, so choose it on validation data and then report test metrics for that one value.
Split each class separately
When rows arrive sorted by label, a tail slice puts almost all positives on one side; a per-class split keeps the ratio.
labels = [0] * 16 + [1] * 4 # rows arrived sorted by class
def rates(name, train, test):
print(f"{name:<17} train {sum(train)}/{len(train)} positive, "
f"test {sum(test)}/{len(test)} positive")
print(f"overall positive rate: {sum(labels) / len(labels):.2f}")
rates("tail split", labels[:15], labels[15:])
zeros = [y for y in labels if y == 0]
ones = [y for y in labels if y == 1]
test = zeros[:4] + ones[:1]
train = zeros[4:] + ones[1:]
rates("stratified split", train, test)Example explained
Line 1labels is ordered by class, which is what you get from a database export sorted by status.
Line 2labels[:15] contains no positive row at all, so the model cannot learn the class it is scored on.
Line 3Taking a quarter of each class separately puts both parts at the overall 20% positive rate.
Line 4In scikit-learn this is train_test_split(X, y, stratify=y); shuffling alone still leaves the ratio to chance on small data.
Important notes
Precision is undefined when a model predicts no positives; the script prints 0.00 so the table stays aligned, but tp=0 fp=0 in the counts is what tells you the model never fires.
With 20 hold-out rows every single prediction is worth 5 accuracy points, so report the raw counts and prefer repeated stratified k-fold over one split when data is scarce.
Common mistakes
Quoting 90% accuracy on a 10%-positive set with no baseline: predicting 'no' everywhere already scores 90%, so the number says nothing about the model and can hide recall 0.00.
Scanning several thresholds or models against the test set and then publishing the best test F1: the test set has silently become training data and the reported score is optimistic.
Fitting a scaler, imputer or feature selector on all rows before splitting: test statistics leak into training, the offline metric looks fine and production quietly disagrees.
Try it yourself
Change, predict, then run
Add two more rows to the main report: an always-predict-1 baseline and the model at threshold 0.70. Then say which of the four rows you would ship, given that a missed churner costs ten times a false alarm.
Open the Python workspaceCheck your understanding
A fraud model reaches 97% accuracy on a test set where 3% of rows are fraud, and the majority-class baseline reaches 97% too. What have you learned?
- The model is strong: it gets 97 out of every 100 predictions right.
- Nothing yet — it matches the always-predict-negative baseline, so it may catch zero fraud cases.
- The model is overfitting, because accuracy that high is only possible on training data.
- Precision must also be about 0.97, since precision is derived from accuracy.
Show answer
Matching the baseline means the model added no information; a constant predictor scores exactly 97% here, and its recall is 0.00, so you must look at tp and fn before claiming anything. Option 0 is the trap: 97% sounds high but is free on this class balance, and precision is computed only over predicted positives, so it can be undefined or terrible while accuracy stays at 97%.