Bias-Variance and Error Analysis

Interviewers write two numbers and wait. Model A: train error 0.11, test error 0.13. Model B: train 0.02, test 0.31. Model C: train 0.18, test 0.47. Thirty seconds of arithmetic separates candidates who memorized slogans from candidates who have trained models.

The decomposition you are approximating: under squared loss, expected test error = bias-squared + variance + noise. Bias-squared is how far the model class sits from the true function even with infinite data; variance is how much the fitted function moves when the training sample changes; noise is irreducible label randomness. Train and test errors are not those terms exactly - training error is measured on data the model has seen, so it flatters both - but the pair gives usable estimates:

Model A cannot even fit the data it has seen: more regularization or a fresh split will not rescue it, it is too simple for the signal or the features do not carry the target. Model B has near-perfect training fit and the world disagrees - textbook variance. Model C is bad on both counts, which usually points at the pipeline itself: leaky features, mismatched preprocessing between train and test, or a drifting distribution, all worth auditing before any architecture work.

The first intervention depends on which column is bad:

A notebook-sized diagnostic you can paste anywhere:

def diagnose(train_err: float, test_err: float) -> str:
    gap = test_err - train_err
    if gap > max(0.05, 2 * train_err):
        return f"variance-led (gap {gap:.2f}): regularize or add data"
    if train_err > 0.15:
        return f"bias-led (train {train_err:.2f}): add capacity or features"
    return f"healthy (gap {gap:.2f}, train {train_err:.2f}): tune and verify"

Two caveats keep you honest. A large gap is not uniquely overfitting - distribution shift produces the same signature, so re-run evaluation on a time-ordered split: if the gap grows toward the present, you are looking at drift, not variance. And error rate hides confidence: a classifier can be right on 92% of rows while assigning 0.51 to the ones it gets right, so track cross-entropy alongside accuracy; the gap is variance, the level is bias, and the split strategy you used decides how much to trust both. The fold standard deviation you produced in Cross-Validation Strategies measures spread across splits, which is a cousin of the spread across samples measured here. And one final discipline: record both numbers, not just the test error. A test error of 0.13 alone tells you nothing about the intervention; with the training error beside it, the next experiment writes itself.

Roadmap review: we can buy five times more labeled data next quarter - will accuracy go up? The answer you give with a learning curve beats the one you give with a hunch, and the curve takes ten minutes to build.

A learning curve plots train and validation performance as the training set grows: fit the full pipeline on 10%, 25%, 50%, 75%, and 100% of the data, evaluate each on held-out folds, and watch two lines plus the gap between them.

from sklearn.model_selection import learning_curve

sizes, tr, va = learning_curve(
    GradientBoostingClassifier(n_estimators=200), X, y,
    train_sizes=[0.1, 0.25, 0.5, 0.75, 1.0], cv=5,
    scoring="neg_log_loss", n_jobs=-1)

for s, t, v in zip(sizes, tr.mean(1), va.mean(1)):
    print(f"{s:5d}  train {-t:.3f}  val {-v:.3f}  gap {v - t:.4f}")
#  600  train 0.214  val 0.331  gap 0.117
# 1500  train 0.221  val 0.298  gap 0.077
# 3000  train 0.226  val 0.271  gap 0.045
# 4500  train 0.229  val 0.259  gap 0.030
# 6000  train 0.231  val 0.252  gap 0.021

Those five rows answer the question: training error barely moves because the model already fits, validation error keeps falling, and the gap is still closing at full size. Data is the binding constraint, and five times more labels should buy roughly another 0.02 to 0.04 on this metric. Run the same experiment on a different problem and the picture inverts:

Three details make the curve trustworthy. Plot a loss rather than accuracy: accuracy saturates and hides improvement the model is still making, while log loss keeps moving. Compute each point inside the same fold scheme you chose in Cross-Validation Strategies, otherwise every point is a single noisy split and the curve wiggles for reasons that have nothing to do with data size. And extrapolate honestly - the curve assumes new rows come from the same distribution at the same label quality, and five times more data from a different annotator pool can bend it back up.

Carry the curve into the decision as arithmetic: if the validation loss is still dropping about 0.015 per doubling of data and the roadmap needs 0.05, then two more doublings (4x the labels) roughly close the gap, and you can quote that against the price per label. If the curve has flattened by 0.002 per doubling, five times the data buys about 0.005, and the honest recommendation is to spend the budget on features or labels quality rather than label volume.

The same experiment with model size on the x-axis gives the capacity curve: a U shape where the left arm says grow the model and the right arm says shrink it, with double descent as the reminder that past a critical width test error can fall again. Either axis works; the discipline is choosing by validation evidence rather than folklore. The next problem is that your curves are made of runs, and runs are what Experiment Tracking Tools exist to organize.

Your sentiment model sits at 91.0% and the roadmap asks for 95%. The weak answer is try a larger model, which guesses. The strong answer is an error analysis that produces a ranked table in a week.

Pull a random sample of 100 misclassified validation examples - random, not the most dramatic ones, because cherry-picked errors inflate the exotic categories. Write a taxonomy of six to eight buckets before tagging, tag every example, then have a second reviewer tag 30 of them so you can check that two people using the taxonomy agree.

Suppose the counts come back like this:

The arithmetic that makes this table actionable: category share times current error rate equals the maximum points available if you solved that category perfectly. At a 9% error rate, a bucket holding 24% of errors is worth 0.24 x 9 about 2.2 points - the entire roadmap target needs two such buckets, not a new architecture. Buckets with no ceiling cap everyone: if 15% of errors are genuinely double-labelable, 100% is unreachable and 95% may already sit near the label-agreement limit.

A confusion matrix routes you to the same list from the class side:

Neutral recall is 0.81 versus 0.90 for negative: the mass sits in neutral, so that is where the fixes belong. The matrix also exposes asymmetric costs you may not have chosen deliberately: positive predicted as neutral (55) versus neutral predicted as positive (31). If a missed positive costs five times a false alarm, those two cells deserve different thresholds, which is the same policy question a stakeholder will eventually ask you.

Also check whether some errors are policy rather than learning - sweep the decision threshold before touching the model:

import numpy as np
from sklearn.metrics import f1_score
for t in np.arange(0.30, 0.80, 0.05):
    print(f"threshold {t:.2f}  f1 {f1_score(y_val, proba >= t):.3f}")

If a threshold move recovers half the gap, that is a one-line change with no retraining and no new data.

Close the loop: apply the top fix, re-run on the same folds so the comparison stays paired, and record the delta as its own run in Experiment Tracking Tools where the config and data version travel with it. Those buckets then become the known-limitations section of the artifact you will assemble next - interviewers remember candidates who answer how would you get to 95%? with sampling, tagging, and ceilings instead of model names.