Early Stopping and Learning Rate Schedules

You train a three-layer MLP on 40,000 rows for 200 epochs. Training loss walks down to 0.08 and never stops. Validation loss reaches 0.31 at epoch 47 and creeps upward from there — every extra epoch makes the model worse on data it has never seen, and you are paying four seconds an epoch to watch it happen. When a teammate asks how many epochs to train for, the honest answer is 47. Early stopping is the mechanism that finds that number while the job runs, not from a post-mortem plot.

The rule has three parameters: a monitored metric (val_loss), a patience (how many consecutive evaluations without an improvement of at least min_delta before you stop), and a policy for what to do at the stop.

from tensorflow import keras

cbs = [
    keras.callbacks.EarlyStopping(
        monitor='val_loss', patience=12, min_delta=1e-3,
        start_from_epoch=25, restore_best_weights=True),
    keras.callbacks.ModelCheckpoint('best.keras', save_best_only=True),
]
history = model.fit(X_tr, y_tr, validation_split=0.2,
                    epochs=500, batch_size=128, callbacks=cbs)

patience=12 means the loop tolerates 12 evaluations that fail to beat the best-so-far by 1e-3 before halting; start_from_epoch=25 suppresses stopping while the curve is still in its steep descent, where a single flat epoch means nothing. Patience must exceed the noise in your validation metric, or you stop on a wiggle: with a 500-example validation set, the per-epoch fluctuation in loss is often larger than the improvements you are trying to detect, so patience of 2 or 3 is a coin flip. The PyTorch loop shows the same rule without sugar:

import copy, torch

best_val, bad_epochs = float('inf'), 0
best_state = None
for epoch in range(500):
    train_one_epoch(model, train_loader, opt)          # model.train(): dropout active
    val = evaluate(model, val_loader)                  # model.eval(): dropout off
    if val < best_val - 1e-3:
        best_val, bad_epochs = val, 0
        best_state = copy.deepcopy(model.state_dict())
    else:
        bad_epochs += 1
        if bad_epochs >= 12:
            break
model.load_state_dict(best_state)   # ship epoch 47, not epoch 159

That last line is the detail that gets skipped. best_state only exists if you snapshot it; best.keras via ModelCheckpoint is the same idea on disk. The epoch you stop at is rarely the epoch you should deploy — patience extends training past the minimum, so restoring the best weights is the whole point of the exercise.

Why stopping counts as regularization: the validation set is choosing the stopping time, so the model never gets enough gradient steps to finish memorizing the training noise. Complexity grows with every epoch (weight norms drift upward as the fit tightens), and the stop caps that growth. This is also why early stopping and an explicit penalty solve the same problem from two directions, which you will see directly in Regularization with L1, L2, and Dropout.

Two hygiene rules. First, the validation split must mimic deployment: shuffled rows for a random-arrival problem, a strict time cutoff for a forecasting problem — a shuffled split on temporal data makes epoch 47 optimistic. Second, patience and min_delta are hyperparameters you are fitting on validation data; a separate untouched test set still exists for the final number, the same discipline you applied in Grid and Random Search.

Gradient-boosted trees take the identical trick per boosting round instead of per epoch, which is how n_estimators=5000 becomes safe:

model = xgb.XGBRegressor(
    n_estimators=5000, learning_rate=0.05,
    early_stopping_rounds=50, eval_metric='rmse')
model.fit(X_tr, y_tr, eval_set=[(X_val, y_val)])

Same mechanism, one scale down: early_stopping_rounds=50 tolerates 50 flat rounds, then training halts and model.best_iteration names the round to predict with. In an interview, "when do you stop training?" is answered with patience, min-delta, restore-best-weights, and a split that matches production — not with a fixed epoch count you remembered.

Watch the loss curve of a long training run and you see two regimes. For the first epochs, big steps carry the weights from random initialization into a good basin. Then the curve flattens into a floor that never quite reaches zero, and the weights jiggle: with a constant learning rate η, SGD takes steps whose size scales with η times the gradient noise, so the iterate orbits the minimum at a radius proportional to η instead of settling into it. The fix is not more epochs. A constant learning rate is a choice to stop refining exactly when refining starts to matter, and a schedule is the correction.

A schedule replaces the scalar η with η(t), where t can be epochs, batches, or iterations:

Cosine annealing is the workhorse because it spends its largest steps early and crawls through the last 10% of training at near-η_min, polishing the solution:

import torch

opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=60, eta_min=1e-4)

for epoch in range(60):
    train_one_epoch(model, train_loader, opt)
    sched.step()                       # once per epoch, after the epoch's updates
print([g['lr'] for g in opt.param_groups])

A scheduler you never step is a constant learning rate, and the most common bug in the code above is stepping it once after the loop — 60 epochs at 0.1 — or stepping per batch against a schedule whose T_max counts epochs, which decays 128 times faster than intended. When in doubt, print opt.param_groups[0]['lr'] at epoch 0, 30, and 60 and confirm the numbers you expect.

Keras expresses the same policy as a callback, and the plateau version earns its keep by combining with early stopping:

reduce = keras.callbacks.ReduceLROnPlateau(
    monitor='val_loss', factor=0.5, patience=5,
    min_lr=1e-6, verbose=1)
stop = keras.callbacks.EarlyStopping(
    monitor='val_loss', patience=15, restore_best_weights=True)

Read those two numbers together: the learning-rate drop must come before the stop, so patience on the reducer (5) has to be smaller than the stopper's (15). With both at 10 you stop the moment the first decay would have paid off — a wasted decay cycle. The reducer also documents itself: verbose=1 prints Epoch 43: ReduceLROnPlateau reducing learning rate to 0.05, which is evidence you can put in a report.

Two schedule facts interviews probe. First, transformers warm up on purpose: with Adam and randomly initialized weights, early gradients are huge and the first updates overshoot wildly, so η ramps linearly over a few thousand steps before decaying — the original Transformer used η_t = d_model^(−0.5) × min(t^(−0.5), t × warmup^(−1.5)). Second, decaying the learning rate changes what early stopping means: once η is small the weights barely move, validation loss flattens, and patience starts expiring for reasons that have nothing to do with overfitting. That interaction — schedules, restarts, and the ways a stopping loop misleads you — is the next chapter's subject.

Cosine annealing has a property that looks like a bug and is actually the feature: it drives the learning rate to almost zero at the end of each cycle. A rate pinned near zero cannot leave the basin it fell into, and basins found late in training tend to be narrow — sharp minima that fit the training set beautifully and generalize worse. SGDR (Loshchilov and Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts") exploits this by cutting the anneal short and jumping back up:

sched = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(
    opt, T_0=20, T_mult=2, eta_min=1e-6)
# cycle 1: 20 epochs, cycle 2: 40 epochs, cycle 3: 80 epochs — η resets to 0.1 each time

The restart is a jump back to the initial learning rate, with T_mult stretching each successive cycle because later cycles are for refinement, not exploration. The high-rate phase gives the iterate enough energy to escape the sharp valley it settled into and find a flatter one; the low-rate phase pulls it to the bottom of that valley. Empirically the final model sits lower on validation loss than a single monotone anneal of the same total length.

The same machinery buys an ensemble almost free. If you snapshot the weights at the end of every cosine cycle, you get k models trained at roughly 1× the cost of one run, because only the last cycle of each snapshot is shared work. Averaging their predictions reduces error according to the classic ensemble relation:

ε_ensemble ≈ ε × (ρ + (1 − ρ)/k)

where ρ is the average pairwise correlation between the members' errors. With k = 4 members at ρ = 0.6, the effective error drops to 0.75ε. The correlation term is why snapshots help less than independently trained models — they share ancestry — but far more than nothing for zero extra training.

Now the ways the loop misleads you. Each of these has bitten a real pipeline:

1. You deploy epoch 159 instead of epoch 47. Patience guarantees the run outlives the minimum. restore_best_weights=True, a ModelCheckpoint(save_best_only=True), or your own best_state snapshot — one of them must exist, and a serving team must be told which.
2. Patience below the metric's noise floor. Compute the per-example standard deviation of your validation loss, σ, on n examples; the epoch-to-epoch standard error is about σ/√n. Improvements smaller than that are noise, and min_delta should sit at or above it. On 500 examples that floor is often larger than you think.
3. Evaluating every batch. Each evaluation is a query against the validation set, and queries are how you overfit it. Evaluate once per epoch on a fixed set, deterministically (dropout off, no random augmentation), so epochs are comparable.
4. Restarting the schedule without restarting correctly. CosineAnnealingWarmRestarts resets η but keeps the optimizer's momentum buffer; if you instead rebuild the optimizer each cycle, you zero the momentum and the "restart" is a limp. Decide which you mean and test that the printed LR actually jumps.
5. Tuning patience, min_delta, or epoch budget on the test set. Same winner's-curse argument you met for hyperparameter search in Grid and Random Search: adaptive queries against a held-out set bias the number you eventually report.

The interaction worth remembering is temporal. Plateau-based decay and early stopping both watch the same curve, so their patience values are a pair, not two independent knobs: give the learning-rate drop a chance to work before you stop (reducer patience < stopper patience, roughly 1 : 3). Otherwise the schedule's final decay fires after the run already ended — you paid for the schedule and collected none of it.

With stopping and schedules settled, the remaining question is why the validation curve sits above the training curve at all. That gap is a capacity-versus-data problem, and the levers for it — penalties, dropout, augmentation — are Regularization with L1, L2, and Dropout.