Regularization with L1, L2, and Dropout

A fraud team gives you 200 candidate features and a mandate: the scoring service must explain every decision by naming the features it used. A plain logistic regression fits all 200 and its coefficients wobble wildly between cross-validation folds — correlated features trade weight back and forth, and last week's top feature is this week's noise. Adding a penalty to the loss fixes both problems at once: the fit gets stable, and most coefficients collapse to exactly zero.

The two penalties differ in one algebraic step, and everything else follows from it.

L2 (ridge): minimize L(w) + λ Σ w_j². The gradient of the penalty is 2λw, so each SGD step becomes:

w ← w − η(∇L + 2λw) = (1 − 2ηλ)·w − η∇L

That factor (1 − 2ηλ) < 1 multiplies the weight down every single step — multiplicative shrinkage. Over N steps of a fixed rate the cumulative multiplier is approximately (1 − 2ηλ)^N ≈ e^(−2ηλN): weights decay exponentially toward zero but, in exact arithmetic, never reach it. L2 makes weights small; it never makes them zero.

L1 (lasso): minimize L(w) + λ Σ |w_j|. The derivative of |w| is sign(w), a constant-magnitude pull of λ regardless of how tiny the weight already is:

w ← w − η·∇L − ηλ·sign(w)

A small weight gets pulled by the same force as a large one, so it crosses zero and is clamped there — the proximal update is soft-thresholding, w_j ← sign(w_j)·max(0, |w_j| − ηλ). L1 produces exact zeros, which is why it doubles as feature selection: 200 features in, 37 non-zero coefficients out, and the fraud team gets its explanation list.

Geometry explains the sparsity without any calculus: level sets of the loss are ellipses, and the feasible region |w₁|+|w₂| ≤ t is a diamond with corners on the axes. The constrained optimum sits where the ellipse first touches the diamond, and because a corner is a sharp point, contact happens at an axis with probability 1 — one weight is exactly zero. L2's circle has no corners, so contact lands anywhere.

In scikit-learn the penalty strength is alpha (for Ridge/Lasso) or C (for LogisticRegression, where C is inverse regularization — large C means weak penalty). That quantity acts multiplicatively, so it belongs on a log scale in any sweep, exactly as in Grid and Random Search:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_validate, StratifiedKFold
from scipy.stats import loguniform

for C in [1e-3, 1e-1, 1e1, 1e3]:           # decades, not 0.1/0.2/0.3
    clf = LogisticRegression(C=C, penalty='l1', solver='liblinear',
                             max_iter=5000)   # liblinear/saga support L1
    cv = cross_validate(clf, X_tr, y_tr, cv=StratifiedKFold(5, shuffle=True,
                        random_state=42), scoring='roc_auc')
    print(f'C={C:>6}  AUC={cv["test_score"].mean():.4f}  '
          f'nonzero={sum((clf.fit(X_tr, y_tr).coef_ != 0).ravel())}')

The loop reports the two axes you actually trade off: validation AUC and the number of surviving features. Pick the smallest model within one standard error of the best AUC.

One distinction interviewers like to probe: L2 and weight decay are the same thing for plain SGD, but not for Adam. With SGD, weight_decay=λ adds λw to the gradient, identical to L2. Adam then divides that gradient by √v (a running second moment), so the effective shrinkage depends on each weight's gradient history — often far weaker or stranger than intended. AdamW applies the decay directly to the weights, decoupled from the adaptive scaling: torch.optim.AdamW(params, lr=3e-4, weight_decay=0.01) versus torch.optim.Adam(params, weight_decay=0.01). In deep learning you almost always want the decoupled form; L1 rarely appears directly in neural nets because exact zeros fight with optimizers' momentum, which is where dropout takes over — the next chapter.

A 512-unit hidden layer trained on 30,000 examples learns a comfortable shortcut: unit 217 and unit 91 fire together on every training batch, and the classifier leans almost entirely on that one path. Training accuracy hits 1.0 in fifteen epochs; validation accuracy stalls at 0.79. Dropout attacks the shortcut directly: during training, each unit is independently zeroed with probability p before it reaches the next layer.

import torch.nn as nn

class Net(nn.Module):
    def __init__(self, d_in=64, d_out=2):
        super().__init__()
        self.fc1 = nn.Linear(d_in, 256)
        self.drop = nn.Dropout(p=0.3)      # p = 0.3 → each unit survives with prob 0.7
        self.fc2 = nn.Linear(256, d_out)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.drop(x)                   # active only while model.train()
        return self.fc2(x)

Every training forward pass therefore evaluates a different thinned subnetwork — with n units there are 2^n possible masks, and gradient updates accumulate across all of them. The weights can never rely on unit 217 and unit 91 co-firing, because in roughly half the batches one of them is absent. Robustness to a missing partner is learned instead.

Two mechanics questions come up every time, and both have crisp answers.

Why scale at training time? PyTorch and Keras both use inverted dropout: activations are multiplied by 1/(1 − p) during training, so the expected activation each unit sends downstream is unchanged, and inference runs the full network with raw weights — no adjustment. The legacy Theano-style alternative (scale by 1/(1−p) at test time instead) is what breaks models ported between frameworks: a Keras model evaluated inside a hand-written Theano-style loop comes out 1/(1−p) too small. If your predictions change run to run in production, the single most likely cause is that you forgot model.eval():

model.train()   # dropout active, BatchNorm updates running stats
model.eval()    # dropout is the identity, predictions are deterministic
with torch.no_grad():
    preds = model(X_test)

model.eval() does not mean "evaluate the metric"; it means "switch off train-time behavior". Skipping it gives nondeterministic serving predictions that vary by several accuracy points — a bug that survives code review because training code usually calls train() for you.

Why is it an ensemble? With masks m ∈ {0,1}ⁿ drawn with E[m_j] = (1 − p), the dropout output at expectation equals the output of the geometric mean over subnetworks, so training optimizes an average over 2^n models that share weights — Hinton's description is "an efficient approximation to averaging over all thinned networks." Gal and Ghahramani proved the train-time mask corresponds to a variational approximation over the model's posterior, which is why Monte Carlo dropout at inference works: run K=10 forward passes with dropout enabled under torch.no_grad(), average them for the prediction, and take the variance as epistemic uncertainty — high variance means the ensemble disagrees, i.e. the input is unlike anything it saw.

Where dropout is the wrong tool: convolutional networks with BatchNorm, where zeroed activations corrupt batch statistics and augmentation already supplies most of the regularization; and tiny models where the mask adds variance to an already-noisy gradient. Where it is the right tool: wide fully-connected towers trained on modest data — precisely the model that memorizes through one pathway.

Do not stack every lever at once. Dropout 0.5 + weight decay 1e-2 + aggressive augmentation + short patience on a 20,000-row dataset produces a model whose training loss never gets below 0.9: that is underfitting, and the fix is removing regularization. Knowing which lever to reach for, and in which direction, is the diagnosis you will practice in Diagnosing the Generalization Gap; the next chapter builds the decision table and adds the regularizer that costs nothing — more training examples.

Every training image in your chest-X-ray dataset arrives already centered, and the model scores 0.94 on validation — then a hospital sends a film shifted two centimeters left and accuracy falls to 0.71. The dataset did not shrink; the model simply never saw that variation. Data augmentation regularizes by changing what the model is allowed to memorize: if every epoch presents a different plausible rendering of the same label, the only stable thing left to learn is the mapping itself.

The arithmetic of augmentation is why it beats most hand-built tricks. A 32×32 CIFAR image has 32 valid horizontal crops × 2 flip states = 64 distinct training views per stored image; 50,000 images become 3.2 million views a year of training without a single byte of new data. The standard torchvision recipe:

import torch, torchvision.transforms as T

train_tf = T.Compose([
    T.RandomCrop(32, padding=4),
    T.RandomHorizontalFlip(),
    T.ColorJitter(brightness=0.2, contrast=0.2),
    T.ToTensor(),
    T.Normalize((0.4914, 0.4822, 0.4465), (0.2470, 0.2435, 0.2616)),
])
val_tf = T.Compose([T.ToTensor(), T.Normalize(...)])   # NEVER augment at evaluation
criterion = torch.nn.CrossEntropyLoss(label_smoothing=0.1)
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05)

Two lines in that block deserve attention. Label smoothing replaces the one-hot target with (1 − ε)·y + ε/K — with ε = 0.1 and K = 10, the target is 0.91 for the correct class. Its effect is mathematical: a plain cross-entropy keeps pushing the correct logit toward +∞ even after the prediction is right, driving confidence and weight norms upward forever; the smoothed target caps the optimum at a finite logit of roughly log((1−ε)/(ε/K)), so the model stops inflating its confidence. Calibrated probabilities and a smaller weight norm come out of the same change. Second, weight_decay=0.05 belongs on AdamW, because (as covered above) decoupled decay is what actually shrinks the weights — and standard practice excludes biases and normalization parameters from decay while applying it to the 2-D weight tensors.

More data variants that act as regularizers:

Now the decision problem you will actually face in an interview: the generalization gap is open — training loss 0.12, validation 0.48 — and you have a drawer full of regularizers. Which one, and in which direction? Read the train metric first, because the wrong direction actively harms:

The stop-at-k row in that table is not a manual decision either — the patience rule from Early Stopping and Learning Rate Schedules automates it, and the rest of the levers above change what the curve looks like before the stopper fires.

Stacking is a search, not a checklist: λ (weight decay), p (dropout), ε (smoothing), and augmentation strength are four hyperparameters interacting on one axis of "how constrained is the model". Tune them jointly on a log scale with the budget arithmetic from Grid and Random Search, or let a surrogate spend your budget more cleverly — that is exactly what Bayesian Hyperparameter Optimization does with Optuna.

One training-loop hygiene point that produces phantom over-regularization reports: excludes bias and BatchNorm parameters from weight decay, or at least know that you are not doing it. In PyTorch the default AdamW(model.parameters(), weight_decay=0.05) decays everything, including the shift and scale parameters whose whole job is to be free:

decay = [p for n, p in model.named_parameters() if p.ndim >= 2]
no_decay = [p for n, p in model.named_parameters() if p.ndim < 2]
opt = torch.optim.AdamW([
    {'params': decay, 'weight_decay': 0.05},
    {'params': no_decay, 'weight_decay': 0.0},
], lr=3e-4)

A quick sanity check before shipping any regularized model: train and validation accuracy must both be plotted. A model with train 0.86 and val 0.85 is healthy; a model with train 0.99 and val 0.83 is not, no matter how good its validation number looks in isolation — and reading exactly that evidence is the next topic.