SGD and Minibatches

The full gradient over n examples is exact but expensive; the gradient of a single example is nearly free but wrong in a specific, quantifiable way. Stochastic gradient descent takes the second option and lives with the consequence: each update uses one randomly sampled training example, whose gradient is a noisy estimate of the true mean gradient, but a noise that averages out over many steps.

Unbiased is the technical claim, and it is worth stating precisely. If g_i is the gradient of the loss on example i, then the expected value of g_i over a uniformly random draw equals (1/n) times the sum of all g_i, which is exactly the batch gradient. No systematic distortion, no bias toward any example; the single-example gradient is centered on the truth. What it carries instead is variance, roughly the average squared deviation of per-example gradients from their mean, and that variance is the entire character of SGD: the trajectory becomes a random walk with drift, wandering around the true descent direction instead of following it exactly.

import numpy as np

rng = np.random.default_rng(0)
X = rng.standard_normal((100, 3))          # 100 examples, 3 features
y = X @ np.array([1.0, -2.0, 0.5]) + 0.1 * rng.standard_normal(100)
w = np.zeros(3)

def grad_i(i):                             # one example's gradient
    err = X[i] @ w - y[i]
    return 2 * err * X[i]

def grad_full():                           # exact mean gradient
    return (2 / len(y)) * X.T @ (X @ w - y)

sample = np.mean([grad_i(i) for i in rng.choice(100, 20, replace=True)], axis=0)
print(np.round(sample, 3))
print(np.round(grad_full(), 3))            # similar direction, different values

The noise is not merely tolerated; deep-learning practice actively exploits it. Because a noisy step can move uphill as easily as downhill, SGD naturally escapes saddle points, where the exact gradient is zero and batch descent stalls. The random perturbation also regularizes slightly, and empirical work shows SGD often generalizes better than full-batch descent on the same network, a phenomenon that has filled its own research literature. And on massive datasets the trade is simply favorable: one example's gradient costs microseconds while the full gradient costs minutes, and a hundred thousand noisy steps cover the loss surface far faster in wall-clock time than a hundred exact ones.

The noise does have a price. Near a minimum, noise keeps the iterate bouncing within a neighborhood whose radius grows with the learning rate and the gradient variance, so the loss never settles exactly at the bottom; the final training loss plateaus slightly above the optimum rather than reaching it. Decaying the learning rate late in training shrinks that neighborhood and lets the trajectory converge, which is why every serious schedule ends with a low-rate fine-tuning phase. The practical question is how to spend the noise budget efficiently, which is exactly what the batch size controls.

A concrete way to feel the variance is to measure it. Continuing the code above, compute every per-example gradient at the same point w, then compare the spread against the mean:

grads = np.array([grad_i(i) for i in range(100)])
mean_g = grads.mean(axis=0)                       # equals grad_full()
std_g = grads.std(axis=0)
print(np.round(mean_g, 3), '\n', np.round(std_g, 3))
# e.g. [-0.264  0.512 -0.131]   mean (the truth)
#      [ 1.912  2.047  1.833]   std  (7x the signal)

When the standard deviation dwarfs the mean, one sample points in a direction barely related to the true one — and that is fine, because ten thousand such samples average back to the truth. Averaging B samples divides that standard deviation by roughly sqrt(B), which is the entire mathematical case for minibatches in the next chapter.

Two interview questions come straight out of this framing. First, "why can SGD escape saddle points but plain gradient descent cannot?" — at a saddle the exact gradient is zero, so batch descent stops, while SGD's per-sample noise gives a nonzero kick that pushes it out along the escaping direction. Second, "is the minibatch gradient an unbiased estimate too?" — yes, for any B, the mean of B sampled gradients has the same expectation as the full gradient; smaller B does not bias the estimate, it only raises its variance.

Textbook SGD samples one example per step; every deep-learning framework you will use implements the minibatch compromise, estimating the gradient from a small subset, typically 16 to 256 examples, and averaging over it. The batch size is a dial between two opposing forces, and understanding both sides is what lets you tune it deliberately instead of copying numbers.

Larger batches reduce gradient variance by the sampling theorem: averaging m independent examples divides the gradient variance by roughly m, so the estimate is closer to the true gradient and the trajectory is smoother. Larger batches also use hardware better. GPUs process matrix products in parallel, and a batch of one leaves thousands of cores idle; batches of 64 to 256 saturate them, so examples-per-second rises steeply with batch size until memory or communication becomes the bottleneck. Two axes, two directions: variance wants small batches, throughput wants large ones.

import torch
from torch.utils.data import DataLoader, TensorDataset

dataset = TensorDataset(X_train, y_train)

# batch size is a loader decision, not a model decision
loader = DataLoader(dataset, batch_size=64, shuffle=True,
                    drop_last=True)      # skip the short final batch

for xb, yb in loader:
    opt.zero_grad()
    loss = criterion(model(xb), yb)      # gradient averaged over the 64 rows
    loss.backward()
    opt.step()

The linear-scaling rule of thumb connects the two: when you multiply the batch size by k, multiply the learning rate by roughly k as well, so the expected step magnitude stays comparable despite the variance drop. Training large batches without warmup is a classic divergence story (the Llama and GPT training writeups both discuss it), because freshly initialized weights receive large, poorly estimated gradients; warmup ramps the rate so early steps stay stable before the linear-scaled rate takes over. Gradient accumulation is the memory-bound escape hatch: split a 256 batch into four 64 microbatches, accumulate their gradients, and step once, getting large-batch math on small-batch hardware.

The wall-clock analysis usually settles the choice. With a fixed compute budget, small batches spend most of their time on gradient noise rather than progress, while very large batches reduce the number of updates per epoch so far that the trajectory cannot explore; the sweet spot is hardware-dependent, often 32 to 256 for vision and 8 to 32 for language models where sequence padding wastes part of every batch. Note also that steps, not epochs, are the true currency: an epoch over 100,000 examples at batch 32 is 3,125 gradient updates, at batch 256 only 390, and comparing schedules across runs requires counting updates rather than passes over data. A run that reports "trained for 10 epochs" without stating the batch size has not told you how much optimization actually happened. One discipline survives every configuration: shuffle before each epoch, which is the next chapter's subject.

What changes concretely when you double the batch but leave the learning rate alone: each step's variance halves, but the step count per epoch halves too, so the trajectory makes half as many noisy moves with the same step size — effective noise per epoch drops by about 4x while progress per epoch stays similar, and the usual symptom is a loss curve that descends smoothly but plateaus early. That is why the linear-scaling rule exists: double the batch, double the rate, and the noisy behavior per epoch is roughly preserved while hardware throughput improves.

For interviews, be able to state the two failure signatures in both directions. Batch too small: gradient estimates so noisy that the loss oscillates visibly and the learning rate you need caps how fast you can descend; also poor GPU utilization, low examples-per-second. Batch too large: fewer updates per epoch, wasted memory on activations, and the well-documented generalization gap — at equal epoch counts, very large batches often land in sharper minima, and matching performance requires not just more learning rate but more total steps or a longer schedule. "Batch size is a hyperparameter of the optimizer, not of the dataset" is a sentence worth being able to defend with both halves of this trade.

A DataLoader with shuffle=True reorders the dataset every epoch, and that default hides a real failure mode that appears the moment it is switched off. Training data in production tables is almost never in random order: events arrive sorted by timestamp, by user id, by store, or by sensor. Feeding that order to SGD without shuffling means each batch is drawn from a narrow slice of the distribution, and the gradient sequence stops resembling the unbiased estimator the theory assumes.

The failure has two forms. With ordered data, the model first spends many updates specializing on whatever occupies the head of the file (last winter's transactions in a time-sorted log), then suffers a distribution shift as the next block arrives, and the loss curve shows sawtooth oscillation synchronized with the data's structure. With class-blocked data, where positives precede negatives, consecutive batches may contain almost one class each, so per-batch gradients point in wildly different directions and the effective learning rate per class spikes. Both are avoidable with one line.

from torch.utils.data import DataLoader

# correct: reshuffle every epoch, independently of model state
loader = DataLoader(dataset, batch_size=64, shuffle=True,
                    generator=torch.Generator().manual_seed(42))

# reproducible shuffling without giving up randomization
for epoch in range(10):
    for xb, yb in loader:      # order differs each epoch by construction
        train_step(xb, yb)

Two subtleties separate correct shuffling from cargo-cult shuffling. First, seed the shuffle generator explicitly when reproducibility matters; shuffle randomness is separate from model initialization randomness, and fixing one while leaving the other free still prevents run-to-run comparison. Second, complete shuffling has cost on very large datasets: shuffling 100 million examples in memory is expensive, so production loaders shuffle in shards, randomizing shard order plus within-shard order, which preserves most of the statistical benefit at constant cost. PyTorch's RandomSampler with a generator, and TensorFlow's default behavior, implement exactly this class of scheme, and both expose the seed so a failing run can be replayed batch for batch.

Time-ordered data adds a judgment call you already know from the splitting topic. Shuffling is correct within the training window, but the validation and test sets must stay chronological because deployment reads data in time order. The apparent contradiction resolves cleanly: shuffle the data the model learns from so gradients approximate the population mean, keep the data you evaluate on in its natural order so the estimate matches deployment. Stated as an interview answer, this distinction, shuffle for training honesty, order for evaluation realism, is a reliable signal that you have actually run training loops rather than read about them. With noisy gradients and shuffling settled, the last topic of this phase examines where noisy trajectories struggle: the Optimization Landscape.

The diagnosis is easy to state once you have seen it: a loss curve whose oscillation period matches the structure of your file order — one cycle per year of data, one per product category block — is almost certainly an unshuffled-loader bug, not an optimizer problem. The debugging move is to log a few sample identifiers per batch; if consecutive batches share a user or a month, the loader is streaming the table in order. A quick unit test asserts that two epochs of batch indices are not identical, which catches the regression before a training run silently wastes a GPU-day.

One honest caveat closes the topic: shuffling guarantees independence between batches only if the dataset itself is sampled i.i.d. from the target distribution. If your data collection biased the log — one crawler, one region, one season — no amount of shuffling fixes it, because every gradient, noisy or not, is estimating the wrong population mean. Shuffling solves order effects; data collection solves distribution effects, and interviewers often use this pair of questions together to see whether you know which lever addresses which problem.