Confusion Matrix, Precision, and Recall

Your spam filter ships and the first thing you report is 96% accuracy. The next thing you hear is that a colleague's legal notice landed in spam. Both facts come from the same four counts, and until you separate them you cannot defend either the model or the metric. Start with the raw numbers: the mailbox holds 1,000 emails, 100 of them truly spam. Your model flags 120 as spam, and 90 of those flags are correct.

Every email lands in exactly one cell of the confusion matrix, a 2×2 table whose rows are the true labels and whose columns are the predictions:

The arithmetic is trivial and worth doing out loud: 100 true spam minus the 90 caught leaves FN = 10 missed spams. The 120 flags minus the 90 correct ones leaves FP = 30 real emails wrongly condemned. The remaining 870 clean emails the model left alone are the true negatives. The four numbers reconcile: 90 + 10 + 30 + 870 = 1,000.

Two vocabulary traps cost people offers, so fix them now. False positive = Type I error = a false alarm: the model cried wolf. False negative = Type II error = a miss: the wolf walked through. A memory hook that survives pressure: the letter names the prediction's claim — a "false positive" claims positive and is false. The positive class is a choice, not a property of the data: call spam positive and the matrix looks like the table above; call ham positive and every cell swaps meaning. Always state which class you made positive before quoting a number, because precision and recall silently follow that choice.

Accuracy is the diagonal over the whole table: (90 + 870) / 1000 = 96%. Now the uncomfortable comparison: a model that guesses "ham" for every email scores 900 / 1000 = 90%. Your sophisticated classifier beat the laziest possible baseline by six points, and the matrix already shows why — accuracy weights the 870 true negatives so heavily that anything happening in the spam rows barely moves it. That gap is the whole reason the next chapter exists. Before leaving the table, map it to the complaints you will actually hear: "your filter ate my concert tickets" is the FP cell, "my inbox is full of phishing" is the FN cell, and "it let one through" is a specific FN you can attach to a row of features. Every stakeholder complaint names a cell — the skill is translating it back to TP/FP/FN/TN before you answer.

scikit-learn returns this table directly, with one orientation gotcha:

from sklearn.metrics import confusion_matrix

confusion_matrix(y_true, y_pred)      # rows = true, cols = predicted
# array([[870,  30],        <- row 0: actually ham  -> [TN, FP]
#        [ 10,  90]])       <- row 1: actually spam -> [FN, TP]]
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
# ravel() reads row-major: never guess the order, unpack it

Pass labels=[...] when a class is missing from the test set — otherwise the matrix silently shrinks and your column indices lie. For K classes the same logic becomes a K×K table: the diagonal counts correct predictions, and every off-diagonal cell is a specific confusion you can act on. That multi-class reading is exactly what you did in Multi-class Strategies when garden tools' entire row sat in electronics' column. The matrix is the raw evidence; precision and recall are the two projections of it that map onto real costs.

Take the same matrix — TP 90, FP 30, FN 10, TN 870 — and project it two ways. Precision asks: of everything the model flagged, how much was right? Precision = TP / (TP + FP) = 90 / 120 = 0.75. Recall asks: of everything that was truly positive, how much did the model find? Recall = TP / (TP + FN) = 90 / 100 = 0.90. Note which counts each formula ignores: precision never looks at FN or TN, recall never looks at FP or TN. That is why one can look great while the other collapses, and why quoting only one invites the follow-up question you cannot answer.

The two errors behind them have different owners. FP grows when the model is trigger-happy; FN grows when it is timid. Your filter catches 9 of every 10 spams (recall 0.90) at the price of condemning 1 in 4 of its accusations (precision 0.75). Whether that trade is acceptable is not a modeling question — it is a product-cost question, which is the subject of the next chapter.

F1 compresses both into one number using the harmonic mean: F1 = 2PR / (P + R) = 2 × 0.75 × 0.90 / 1.65 = 1.35 / 1.65 = 0.818. An equivalent form worth memorizing because it needs no intermediate rounding: F1 = 2TP / (2TP + FP + FN) = 180 / 210 = 0.818. Why not the plain arithmetic mean? Because the harmonic mean punishes imbalance. A model with precision 1.0 and recall 0.0 (it flags exactly one correct item and nothing else) has an arithmetic mean of 0.5 — which flatters a useless model — and a harmonic mean of 0.0, which correctly executes it. F1 is high only when both components are high, and it ignores TN entirely, which is precisely its job on skewed data.

When the business tells you which error costs more, reach for F-beta, which reweights the harmonic mean: F_beta = (1 + β²)PR / (β²P + R). F2 doubles the weight of recall (missed positives hurt more — disease screening); F0.5 doubles the weight of precision (false alarms hurt more — a spam filter that eats legitimate mail). β is a cost knob in disguise, and naming it as such in an interview lands better than reciting the formula.

In scikit-learn the metrics come from one function with an average switch:

from sklearn.metrics import precision_recall_fscore_support

p, r, f1, _ = precision_recall_fscore_support(
    y_true, y_pred, average=None, zero_division=0)   # per-class vector
pm, rm, fm, _ = precision_recall_fscore_support(
    y_true, y_pred, average="macro")   # unweighted mean of per-class scores
# average="binary" (default) needs pos_label; works only for 2 classes

For multi-class, decide how to average. Macro averages the per-class metrics with equal weight, so a 40-row class counts as much as a 4,000-row one. Weighted weights by support and therefore tracks accuracy's blind spots. Micro pools TP, FP, FN globally first — and in single-label multi-class classification, micro-precision, micro-recall, and micro-F1 all equal accuracy, because every mistake is one FP somewhere and one FN somewhere else. Work the difference once so it sticks: three classes with F1 scores 0.90, 0.50, 0.00 and supports 900, 90, 10 give macro-F1 = (0.90 + 0.50 + 0.00)/3 = 0.467, while weighted-F1 = (0.90×900 + 0.50×90 + 0.00×10)/1000 = 0.860. Same model, two numbers, forty points apart — macro screams because the minority class failed, weighted shrugs because it is rare. Report macro when classes are imbalanced, and say why out loud.

Here is the interview scenario built to expose metric laziness. Fraud is 2% of transactions. Your model predicts "legit" for everything and scores 98% accuracy while catching zero fraud. The interviewer asks: is your model any good? If you hesitate, the follow-up is worse: which metric would you have reported, and why? Have the answer mechanical.

The decision procedure, in the order you should say it. First, name the positive class — fraud, not legit — because every downstream number silently depends on it. Second, ask which error is costlier. A false positive declines a good card payment: the customer is annoyed and maybe churns, cost perhaps a few dollars plus support time. A false negative lets a stolen card run: cost is the transaction value plus chargeback fees, often hundreds. Third, pick the metric that punishes the expensive error. When misses are expensive you optimize recall (or recall at a fixed precision floor — "catch 90% of fraud while keeping false alarms under 200 per day"); when alarms are expensive you optimize precision. When both matter you either use F-beta with β set by the cost ratio, or, better, you report the full confusion matrix at the proposed operating point and let the business owner see both cells. Fourth, remember accuracy belongs only where classes are balanced and errors are symmetric — a controlled benchmark, not production data.

What accuracy cannot tell you, three more things. It says nothing about probability quality: two models with identical matrices can disagree wildly on their 0.51-versus-0.99 predictions, and if anyone will act on the score (ranking a review queue, pricing a bid) you also need log loss or a calibration check. It says nothing about where the errors land: 30 false positives on your best customers versus 30 on fraudsters look identical in accuracy. And it is frozen at one threshold: the same model can trade FP for FN continuously by moving its cutoff, which is why a threshold-free view is the right companion — that is exactly what a ROC curve gives you, and it is the next topic.

A quick worked baseline for the practice of metric selection. Suppose the all-negative fraud model is replaced by one that flags 500 transactions of 10,000, catching 80 truly fraudulent (of 200 total): TP 80, FP 420, FN 120, TN 9,380. Accuracy = 9,460 / 10,000 = 94.6% — worse than the lazy model's 98%. Precision = 80/500 = 16%, recall = 80/200 = 40%, F1 = 0.229. Every threshold metric says the new model is a genuine (if imperfect) detector, while accuracy says it regressed. That inversion — the better model has the lower accuracy — is the single most reliable signal that you are looking at imbalanced data and that accuracy has left the conversation.

Drill the vocabulary once more against the matrix, because interviews probe it under time pressure: precision polices the predicted-positive column, recall polices the actual-positive row, F1 is the harmonic compromise, and the confusion matrix itself is the only artifact that never lies. One reporting habit ties it together: never ship a single number — ship the metric, the threshold it was measured at, and the confusion matrix behind it ("recall 0.40 at threshold 0.05, TP 80 / FP 420"). A number without its operating point cannot be compared, reproduced, or defended in a design review, and it is the fastest way to lose an argument you were otherwise winning. You will now move from quoting one operating point to sweeping every cutoff at once — ROC Curve and Threshold Selection makes that sweep and shows you how to choose the cutoff from costs instead of habit.