Cross-Validation Strategies

You report 94.2% accuracy from your single 80/20 split. A teammate reruns with random_state=7 and reports 91.0%. Both numbers are real; neither is the answer. A single holdout is one draw from a sampling distribution, and on a 1,200-row dataset that distribution is wide enough that the gap between you is split luck rather than modeling skill.

K-fold cross-validation narrows it. Shuffle the data, cut it into k equal blocks, and run k evaluations: each block takes a turn as the test set while the other k minus 1 train the model. Every row is scored exactly once, every row trains on k minus 1 folds, and the reported number becomes a mean with a standard deviation attached.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import KFold, cross_validate

cv = KFold(n_splits=5, shuffle=True, random_state=42)
res = cross_validate(GradientBoostingClassifier(), X, y, cv=cv,
                     scoring=["accuracy", "f1"], return_train_score=True)

print(f"val  accuracy {res['test_accuracy'].mean():.3f} +/- {res['test_accuracy'].std():.3f}")
print(f"train accuracy {res['train_accuracy'].mean():.3f}")   # the gap is your variance signal

Choosing k is a bias-variance trade-off on the estimate itself:

Two subtleties worth saying out loud. First, each fold trains on less data than the final model will have, so the CV mean slightly underestimates the deployed model; expect production to land a little above the fold average. Second, the fold-to-fold standard deviation is the number to argue about: 94.2% +/- 0.4 and 94.2% +/- 3.1 are different claims about the same model, and only one of them is ready for a launch review.

Practical discipline follows. Pin random_state so the fold assignment is reproducible, and reuse the same folds for every candidate you compare. Model A on one split versus model B on another measures the models plus the split; both on identical folds is a paired experiment, and paired comparisons detect small real differences with far fewer samples.

Do the cost arithmetic before you commit to a protocol: 5 folds times 4 hyperparameter configurations times 2 model families is 50 fits, and each fit costs a full training run. When one run takes two hours, that is a hundred-hour model-selection job, so the standard move is cheaper search on subsampled data or 3 folds during tuning, then 5 or 10 folds once for the finalists.

One boundary matters: cross-validation does not give you a test set. Every row trains a model k minus 1 times, so the moment you use CV scores to pick a winner, the winner's score is optimistic. Keep a final untouched holdout - or a later time period - for the single number you quote outside the team, and when data is too scarce to spare one, nested cross-validation (an outer loop estimating performance, an inner loop tuning) restores the separation rigorously.

That is also the harness that keeps the many hyperparameters of Transformer Architecture Overview honest - every architecture claim needs the same rotating folds behind it. Whatever number comes out still carries noise, and Bias-Variance and Error Analysis is how you read that noise.

Two data properties break plain KFold the moment they appear, and each has a dedicated splitter. Using the wrong one does not produce a slightly noisy score; it produces a confidently wrong one.

Class imbalance. Fraud sits at 0.4% of transactions. Shuffle-and-cut KFold can hand one fold two positives and another forty, so on the two-positive fold a single prediction swings recall from 0.0 to 1.0. StratifiedKFold keeps each class proportion identical across folds by distributing each class members round-robin:

from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for tr, te in skf.split(X, y):
    print(y[tr].mean().round(4), y[te].mean().round(4))   # both about 0.004

Interview caveats: stratification assumes test folds should mirror the training distribution - true for classification, misleading when you are deliberately evaluating under drift. Multi-label targets need iterative stratification, not plain StratifiedKFold. And a class with fewer than k members cannot appear in every fold; sklearn will tell you so rather than silently mislead.

Repeated entities. One patient contributes 12 visits, one user contributes 500 events, one video yields 30 clips. Rows from the same entity are near-duplicates: demographics constant, device constant, style constant. Scatter them across folds and the model recognizes the entity instead of learning the phenomenon, so the test score measures memorization. GroupKFold keeps every row of an entity in one fold:

from sklearn.model_selection import StratifiedGroupKFold
sgk = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=42)
for tr, te in sgk.split(X, y, groups=patient_id):
    assert set(patient_id[tr]).isdisjoint(patient_id[te])   # no patient leaks

The assertion above belongs in your test suite: it costs one line and catches the silent failure mode where a pipeline refactor drops the groups argument from split and quietly reverts to row-level leakage. Print the group counts per fold too - a fold holding 3 of the 62 patients tells you how few entities carry that fold's estimate.

What the premium of leakage buys is measurable: take 500 rows coming from 62 patients, score the model with a row-level split, then score it again with patients held out. A move from 0.94 to 0.78 is the memorization premium - what the metric gained by recognizing entities rather than learning the phenomenon. Quote that difference when someone objects that the corrected evaluation made the model worse: nothing about the model changed, the evaluation stopped crediting it for abilities it will not have in production.

Which splitter for which dependency:

The scaffold row has published evidence: on MoleculeNet benchmarks random splits routinely report several points higher than scaffold splits on identical data, because structurally related molecules land on both sides. The general rule is simple to state: identify what one row shares with another, then forbid that shared thing from crossing the split - target value, entity id, timestamp, structure, or geography.

Choose your splitter first, then treat the resulting mean and standard deviation as one measurement. After you have fixed the split design, the next question is whether the resulting number is stable or lucky - that is what Bias-Variance and Error Analysis tells you.

Leave-one-out cross-validation sets k = n: each evaluation holds exactly one row out. Its appeal is training size - the model trains on n minus 1 rows, essentially the data it will finally receive, so bias is near zero. Its cost is n training runs, and its dirty secret is that any two training sets overlap in n minus 2 rows. Those fold errors are so highly correlated that LOOCV's estimate can be noisier than 5-fold. Use it when n is genuinely small (30 experimental samples, each one expensive to spare) and never to squeeze a marginal result into significance.

Leave-one-group-out is the useful sibling: one fold per group, every group evaluated once. With 6 countries and a requirement to report per-country generalization, GroupKFold shuffles groups into folds; LOGO gives the honest per-country number instead. The trade is transparent: with 30 groups you get 30 cheap folds, with 3 groups you get 3 expensive and extremely coarse estimates, and below about 5 groups you should report per-group results individually rather than pretending the average means anything.

LOOCV's variance trap deserves its own intuition. Because the n training sets overlap in n minus 2 rows, a hard-to-fit example poisons nearly every fold at once: it is the test row once and a training row everywhere else, so all n errors move together instead of independently. The standard error of the mean does not shrink like 1 over sqrt(n) the way the run count suggests. Five-fold trades a small bias (each fold trains on 80% rather than n minus 1) for far more independent error estimates, which is why it usually wins.

Time series is where imprecise splits turn into wrong ones. Two failures, escalating:

1. KFold(shuffle=True) trains on 2026 rows to predict 2024 rows - the model has seen the answer key.
2. KFold(shuffle=False) avoids shuffling but still trains on the future whenever the test fold precedes part of the training data (test = rows 0-99, train = 100-999).

The correct shape is an expanding window: train only on history, test on the next block, roll forward.

from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5, gap=7)      # 7-row embargo between train and test
for i, (tr, te) in enumerate(tscv.split(X)):
    print(f"fold {i}: train {tr[0]}..{tr[-1]}   test {te[0]}..{te[-1]}")
# fold 0: train 0..165   test 173..206
# fold 4: train 0..837   test 845..878

The gap argument earns its keep: if your features are 7-day aggregates or the label horizon spans a week, rows adjacent to the cut share information with training rows. Purged cross-validation goes further - drop any training row whose label window overlaps a test row, and add an embargo of extra rows around each cut. In finance this is not pedantry; overlapping labels are exactly how a backtest becomes fiction.

Selection table:

Report the mean with the standard deviation, state the split strategy next to the number, and be ready for the follow-up question: how would your score change under a different split? If you cannot answer, you do not yet know your model's performance - and the follow-on diagnosis of that uncertainty lives in Bias-Variance and Error Analysis.