Train, Validation, and Test Splits

You build a churn model, try three algorithms, pick the one with the best score on held-out data, and report 91% to your director. Then the model ships and achieves 84%. Nothing broke on deployment day; the estimate was simply optimistic, because the data you used to choose the model also graded it. The three-way split exists to keep those two jobs statistically separate.

Each partition has exactly one role. Training data is the only data the model may learn parameters from: gradient updates, tree splits, weight matrices. Validation data is never used for parameter updates; it grades candidate models and hyperparameters, and you repeatedly look at its scores while making decisions (which algorithm, what depth, which learning rate, how many components). Test data answers a single question at the end: how will the chosen pipeline perform on unseen data? It influences nothing during development, and you ideally touch it once.

A common default for medium datasets is 70% training, 15% validation, 15% test. In scikit-learn the split is two calls of train_test_split with fixed random seeds and, for classification, stratification so each partition keeps the class mix:

from sklearn.model_selection import train_test_split

X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.176, random_state=42, stratify=y_temp)
# 0.176 of the 85% remainder is 15% of the whole

Two details make the split trustworthy. First, set random_state everywhere; an unseeded split means yesterday's 91% cannot be reproduced today. Second, stratify for classification targets, because a rare class at 2% prevalence can vanish from a random 15% slice, and a model evaluated on a slice missing its rare class reports meaningless metrics.

The validation set also powers decisions beyond hyperparameters. Early stopping watches validation loss each epoch and halts training when it stops improving, which makes the validation set part of the training loop and means it must never double as test. For the same reason, error analysis on validation predictions (inspecting which examples the model misses) is legitimate development work, while the same inspection on the test set is not. Think of it as a budget: every glance at a partition spends some of its statistical independence, and the test set can only afford one.

For grouped data (multiple rows per user, several samples per device) use GroupShuffleSplit or GroupKFold so one group never straddles train and test; otherwise the model recognizes individuals it has already seen and scores a memorization win as generalization.

Splits also interact with preprocessing. Any scaler, imputer, or feature selector must be fit on training data only, then applied to validation and test. Fit a StandardScaler on everything before splitting and the validation set's mean and variance leak into training: the leak is small but real, and pipelines make it impossible to commit (Pipeline([('scale', StandardScaler()), ('model', clf)]) fit only on X_train). The next chapter follows the case where a random split is not just leaky but flat-out wrong: Chronological Splits.

Random splitting assumes rows are exchangeable draws from one distribution. Time-ordered data violates that assumption, and shuffling it before splitting trains your model on the future and tests it on the past. The offline score inflates, production disappoints, and the team blames the algorithm instead of the split.

Take a demand-forecasting model trained on three years of daily sales. A random 70/15/15 split puts days from March 2024 in training and days from March 2023 in validation. Two things go wrong. First, the model has already seen the seasonal pattern it is supposedly predicting: last March's promo week sits in training while this March's sits in validation, so the model memorizes the calendar instead of learning a mapping that transfers forward. Second, validation scores measure interpolation inside a period you already covered, not the extrapolation production demands.

The fix is to split by time. Train on the earliest window, validate on the next window, test on the most recent one, with cutoffs chosen to mimic how the model will actually be used. A useful rule: make the validation window resemble the gap between training data and production reality. If the model will be retrained monthly on all history, a validation window one month wide with training data strictly before it mirrors deployment exactly:

import pandas as pd
from sklearn.model_selection import TimeSeriesSplit

df = df.sort_values('timestamp')
cut1 = df['timestamp'].quantile(0.70)
cut2 = df['timestamp'].quantile(0.85)
train = df[df['timestamp'] <= cut1]
val   = df[(df['timestamp'] > cut1) & (df['timestamp'] <= cut2)]
test  = df[df['timestamp'] > cut2]

# expanding-window CV for tuning inside the train period
tscv = TimeSeriesSplit(n_splits=4)   # each fold: train past, validate future

Feature engineering amplifies the risk because many time features aggregate history. A rolling 30-day mean computed over the full dataframe before splitting uses future rows to build a training row. Compute rolling features within each split using only rows strictly before the row being predicted, and shift targets explicitly: to predict day t, the features may use data through t-1, never through t. A classic interview example is predicting stock movement with a feature that includes same-day close; the model reads the answer and reports near-perfect accuracy.

Normalization has the same trap in disguise. Fitting a StandardScaler or computing target encodings (mean of y per category) over the full time-sorted dataset lets future statistics shape training rows. Both belong inside each fold: compute category means from the training window only, then apply them to validation rows, mapping categories unseen in training to a global fallback value. If your evaluation pipeline cannot answer with certainty that no feature saw data after its prediction timestamp, assume leakage until proven otherwise.

When a single chronological split feels too brittle (one unlucky cutoff), TimeSeriesSplit gives expanding-window cross-validation: fold k trains on the first k blocks and validates on block k+1, always past to future. What you never do with time data is k-fold with shuffling, and for the same reason you never compute target statistics (mean encoding, target leakage-prone aggregations) over the full dataset. If your evaluation pipeline cannot answer with certainty that no feature saw data after its prediction timestamp, assume leakage until proven otherwise.

A 15% test slice of a 10-million-row dataset is 1.5 million rows: massive, overkill, and cheap to shrink. A 15% slice of a 400-row medical dataset is 60 rows, and its accuracy can swing by 8 points between two reasonable splits. The right test size is not a percentage; it is whatever makes the score stable enough for the decision you need to make.

For classification, the test accuracy on n independent examples is a binomial proportion with standard error sqrt(p(1-p)/n). At p = 0.9, a 60-row test set gives a standard error of about 3.9 points, so a reported 91% is really somewhere in the 83-99% range at 95% confidence. Quoting that number without an interval overstates your knowledge. Doubling precision requires quadrupling the test set: to cut the error bar in half you need 4x the rows.

import numpy as np
from sklearn.model_selection import (train_test_split, cross_val_score,
                                     StratifiedKFold, KFold)

# small dataset: skip the fixed validation set, use k-fold for tuning
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring='f1')
print(scores.mean(), scores.std())   # e.g. 0.812 0.031

# regression with no ordering concern
kf = KFold(n_splits=5, shuffle=True, random_state=42)

Cross-validation is the standard remedy for small data. Five-fold stratified CV trains on 80% per fold and rotates the 20% holdout five times, giving a mean and a spread from one dataset. The spread matters as much as the mean: a model averaging 0.81 with folds ranging 0.74 to 0.88 is less convincing than one averaging 0.80 within 0.79-0.81. Leave-one-out (k = n) is the extreme, useful only when data is tiny and each fit is cheap, because its variance across folds can be high and its compute cost is n fits.

Budgeting splits by purpose keeps the roles clean. With plenty of data, 70/15/15 is comfortable, and you can shrink the test slice to 5-10% once it holds tens of thousands of examples. With little data, hold out a 15-20% test set, run all tuning with k-fold CV on the remaining 80-85%, and accept wider error bars; report scores as mean plus/minus standard deviation instead of a single heroic number. With very little data, use nested cross-validation: an outer loop estimates generalization, an inner loop tunes hyperparameters, so the estimate never sees the tuning decisions.

One discipline survives every configuration: a test set consulted repeatedly stops being a test set. Each time a team looks at the test score and reacts (tweak features, drop a column, rerun), the test set becomes another validation set and the final number drifts optimistic. Track every test-set evaluation in the experiment log; if the count exceeds one or two, rerun with a fresh holdout before you quote the result to stakeholders.

A worked sizing example ties the pieces together. Suppose stakeholders need to distinguish a 92% model from an 88% model before choosing a vendor. The decision needs a confidence interval narrower than plus or minus 2 points; with p near 0.9, n = 0.9 * 0.1 * (1.96 / 0.02)^2 gives roughly 865 independent test examples. If the dataset holds 3,000 rows, a 30% test slice suffices and the remaining 70% runs five-fold tuning. Notice the direction of the reasoning: the decision defines the precision, the precision sizes the test set, and whatever remains trains the model. That discipline is what the ML Project Lifecycle formalizes next.