Vision Model Error Analysis

Your camera-trap classifier reports 88% validation accuracy and the demo goes well. Then a wildlife biologist asks one question: "how are wolves doing?" You do not know — accuracy is a single number over 900 images and it answers nothing about any class you actually care about. Build the confusion matrix and the model's real behavior appears.

A 4-class run (rows = true label, columns = predicted):

792 of 900 correct = 88.0%, yet wolf recall is 49.4% — half the wolves are being called dogs, and they are not scattered: 30 of 43 wolf errors land in one column. That asymmetry is invisible in accuracy and is the single most actionable fact you have extracted from the model so far. It also tells you precision separately: wolf precision is 42/(42+1+8+7) = 72.4%, so when the model does say wolf it is mostly right — it is missing wolves, not inventing them. If your product counts wolves, recall is the metric; if it flags poaching alerts, false positives dominate the cost. The confusion matrix separates those two failure modes; accuracy cannot.

from sklearn.metrics import confusion_matrix, classification_report

y_true = [...]   # 900 ground-truth labels
y_pred = [...]   # 900 model predictions

cm = confusion_matrix(y_true, y_pred, labels=["rabbit", "fox", "dog", "wolf"])
print(cm)
print(confusion_matrix(y_true, y_pred, normalize="true").round(3))   # row-normalized
print(classification_report(y_true, y_pred, digits=3))

Always print both forms. Raw counts show where the volume is (30 dog-predictions for true wolves dominates the error budget). Row-normalized counts show rates, which is what you compare across classes of different size — the raw matrix makes small classes look harmless simply because they are rare. classification_report gives per-class precision, recall, and F1; macro-averaged F1 is the number to quote when classes are unbalanced, because it weights a failing wolf class equally with a plentiful rabbit class.

You can reproduce the table above without sklearn in four lines — useful when you are explaining the mechanics in an interview:

import numpy as np
cm = np.array([[300, 6, 3, 1],
               [12, 210, 15, 8],
               [4, 9, 240, 7],
               [2, 11, 30, 42]])
row = cm / cm.sum(axis=1, keepdims=True)        # divide each row by its true total
print(np.diag(row).round(3))                    # [0.968 0.857 0.923 0.494] = recall

Know which averaging your metric uses. Micro-averaging pools every decision across classes into one count — for single-label problems it collapses to plain accuracy, so it hides the wolf again. Macro-averaging computes each class's F1 and averages those, giving every class one vote regardless of size. Reporting all three (accuracy, macro-F1, worst-class recall) makes it impossible to lose the failure mode that matters while everyone argues about the headline number.

Interview framing: "your accuracy is 88%, what do you do next?" The answer that lands is a sequence, not a model change — read per-class recall, find the largest off-diagonal cells, name a hypothesis for why that pair confuses (wolves and dogs genuinely share coat and shape features), then verify the hypothesis with images. The matrix localizes the problem; it does not explain it. Explaining it means looking at actual pictures in groups, which is the next section — and the same numbers will be re-cut after the fixes you design with Data Augmentation and Transfer Learning.

The confusion matrix tells you which class pair is broken. Slicing tells you under what conditions, and conditions are what you can change — more night images, a flip-free augmentation, a higher-resolution input. The workflow is: collect errors, group them into slices, compute each slice's error rate, and test one hypothesis at a time.

Start by dumping every misclassification with enough context to sort it:

import torch, json

records = []
with torch.no_grad():
    for paths, x, y in val_loader:                # paths preserved in the loader
        logits = model(x)
        prob = torch.softmax(logits, 1)
        conf, pred = prob.max(1)
        for p, t, pr, c in zip(paths, y, pred, conf):
            if t != pr:
                records.append({"path": p, "true": int(t), "pred": int(pr),
                                "conf": round(float(c), 4),
                                "brightness": float(x.mean())})

json.dump(records, open("errors.json", "w"), indent=2)
print(len(records))                                # 108 errors to inspect

Now group — some slices come from your annotations, others you compute from the images themselves:

Two practices separate rigorous error analysis from scrolling through a gallery. First, view errors as contact sheets — grids of thumbnails, 25 per page — because humans spot recurring patterns in grids (all dark, all with water in frame, all with a fence) that a list of filenames will never surface. Second, sample a control set of correct predictions at the same rate: you will find night images the model got right too, and comparing the two groups tells you whether night is truly the driver or just correlated with being underrepresented. Slicing on a variable the model never saw (time of day was not an input) still works — you are slicing the data distribution, and a slice where error concentrates is a distribution fact worth fixing even when you do not understand the mechanism.

Then convert each slice into an action, with a named owner and a measurable prediction:

• Night error 31% -> brightness/contrast jitter plus night-specific capture data -> predict night error drops below 15% next run.
• Wolf -> dog at 0.93 confidence -> expert image review before touching the model (next chapter) -> if labels are fine, add hard negatives to training.
• Small objects -> raise input resolution or improve crops -> predict errors on <40 px slice drop by half.

Rank those actions by expected value, because you will not ship all of them: expected gain = slice size x achievable error reduction. Dropping the night slice from 31.1% to 15% saves 132 x 0.161 ≈ 21 images, and 21 of 900 is +2.3 points of overall accuracy — a defensible week of work. Fixing the wolf -> dog pair entirely (30 images) is worth +3.3 points but is likely not fixable to zero; halving it is +1.7. A large slice with an unclear fix loses to a medium slice with a cheap one, and writing the arithmetic down is what keeps the iteration honest.

That last column is what makes error analysis scientific instead of anecdotal: every pass through the loop ends with a falsifiable prediction for the next evaluation, and you keep the log so a reviewer can audit why you spent the week on night captures rather than a bigger backbone. The most uncomfortable branch of that loop — when the expert says the model was right — is the next chapter.

You hand the 61 high-confidence errors to the biologist with one instruction: for each image, mark the verdict — model wrong, label wrong, or genuinely ambiguous. She comes back with 13 of them mislabeled, 38 genuinely wrong, and 10 cases where she argues the classes themselves are ill-defined (a fawn-colored dog at 40 pixels is simply not adjudicable as wolf vs dog). That verdict column is the point of the whole exercise: it decides which of three completely different backlogs your next sprint goes into.

1. Label wrong (13 images) -> a data-quality task, not a modeling task. Worse: those wrong labels are in validation, so your 88% is measured against a broken ruler. Fix eval labels first — until you do, every experiment ranks models against noise — then fix the corresponding training labels before retraining.
2. Model wrong with clean labels (38) -> a modeling task: harder slices from the previous chapter go into augmentation, sampling, or capacity decisions.
3. Ambiguous by construction (10) -> a product task. Either the taxonomy needs an adjudication rule ("if tail height < shoulder, label dog") or the classes need merging, or an "uncertain" output must be an allowed answer. No architecture fixes a boundary that humans cannot draw.

Quantify the human boundary so you stop chasing a ceiling that does not exist. If two experts independently label 200 images and agree on 82%, while their marginal label frequencies put expected chance agreement at 55%, Cohen's kappa is (0.82 - 0.55) / (1 - 0.55) = 0.60 — decent, not perfect. If humans agree at 60% on the wolf/dog boundary, no model can be trusted far beyond it there, and reporting a wolf-vs-dog F1 without that context overpromises. Kappa, not raw agreement, is the number to quote because raw agreement flatters you whenever one class dominates.

The review loop, formalized:

1. Export errors + confidence + slice tags        (errors.json from last chapter)
2. Expert labels each: model-wrong / label-wrong / ambiguous
3. Recount after relabeling -> corrected validation score
4. Merge verdicts into three backlogs with owners
5. Retrain only for backlog 2; re-audit backlog 1 on a fresh sample

Triage the audit queue by model confidence when expert time is scarce. An error the model calls at 0.97 is either a systematic blind spot or a wrong label — high-value either way — while a 0.55 error is ordinary ambiguity that retraining data may resolve on its own. Sort the review sheet by confidence descending and the first page will contain most of your diagnosis.

Two hygiene rules keep it honest. Never tune against the test set — error analysis lives on validation, final numbers come from a test split you touch once per release; and sample audit at a fixed rate every cycle (say 100 images) so the mislabel rate trend is comparable across releases instead of a function of how much effort one reviewer had that day. While you are in the data anyway, run a near-duplicate check across splits — perceptual hashes catch a photo appearing in both train and validation in different crops, which no confusion matrix will ever reveal and which inflates every score on the file.

Expect the first audit to be sobering — double-digit label error rates are common in fast-moving datasets — and also clarifying: half your "model problems" become data problems with owners. When that loop stabilizes, you have squeezed everything you can out of the vision pipeline, and the roadmap pivots to a different input structure where rows are ordered: Sequence Modeling Basics.