Gradient Descent Fundamentals

All of training, from linear regression to a large language model, is one line repeated. At each step, take the gradient of the loss with respect to every parameter, multiply by a small positive number called the learning rate, and subtract: w_new = w_old - learning_rate * gradient. The gradient points uphill, so subtracting walks downhill, and repeating the step from new positions traces a path down the loss surface toward a minimum.

The rule deserves a slow read because every symbol carries a decision. The gradient comes from backpropagation and encodes the surface's local slope. The learning rate encodes how much you trust that local slope: the gradient's promise holds in a neighborhood, and the step size decides how far you travel outside it. The subtraction encodes descent rather than ascent; gradient ascent, used to maximize log-likelihood objectives, flips the sign and is otherwise identical. And the update applies to every parameter at once, which is what lets a single expression govern a model with millions of weights.

Work a concrete trajectory. Loss L(w) = (w - 4)^2 with gradient 2(w - 4), learning rate 0.1, starting at w = 0:

step 0: w=0.0    grad=2(0-4)=-8.0    w_new = 0 - 0.1(-8.0) = 0.8
step 1: w=0.8    grad=-6.4           w_new = 1.44
step 2: w=1.44   grad=-5.12          w_new = 1.952
... converges toward w=4, each step covering 80% of the remaining distance

That geometric decay is not an accident of this example. For a quadratic, gradient descent contracts the distance to the minimum by a fixed factor per step, (1 - 2learning_ratecurvature), so progress is fastest far away and slows asymptotically near the optimum. It also foreshadows the learning-rate story: if the factor's magnitude exceeds one, the distance grows instead of shrinking, which is divergence in one line. The same arithmetic works with a vector of weights unchanged: replace the scalar gradient with the gradient vector, and every weight moves simultaneously by its own slope, which is why one update expression trains a network with millions of parameters.

def gradient_descent(lr, steps=50, w0=0.0):
    w = w0
    trace = [w]
    for _ in range(steps):
        grad = 2 * (w - 4)
        w = w - lr * grad
        trace.append(w)
    return trace

print(gradient_descent(0.1)[-1])    # ~4.0, converged
print(gradient_descent(1.1)[-1])    # oscillates and diverges

The choice of starting point also matters in practice, not just in toy examples. A poor initialization (all-zero weights producing a dead ReLU, or a transformer with an unscaled random init) can place the loss in a region where gradients are near-zero, and no learning rate rescues the run. That is why frameworks ship He or Xavier initialization for neural networks and why transformers use warmup: start small, verify gradients flow, then ramp up.

Two practical properties complete the picture. The update is computed once per pass over the data in its classical form, which you will meet as batch gradient descent shortly. And the rule is oblivious to anything but the current gradient: it carries no memory of previous steps, a limitation that momentum, the next phase's topic, exists to repair. For interviews, being able to run three steps of this arithmetic by hand, and to say why the step shrinks near the optimum, demonstrates that you understand the mechanism rather than the slogan.

The learning rate is the single most consequential hyperparameter in training, and its failure modes are symmetric in opposite directions. Too large, and each step overshoots the minimum, lands on the opposite slope with an even steeper gradient, and the oscillation grows until the loss becomes NaN. Too small, and the model crawls: each step makes negligible progress, training takes unaffordable epochs, and you risk stopping before reaching any useful minimum. Between them lies a workable range that differs per problem, per optimizer, and per architecture.

The mechanism is visible in the quadratic picture. On a loss with curvature c, the distance to the minimum contracts by a factor of (1 - 2lrc) per step. That factor goes negative when lr exceeds 1/c, which is the oscillation threshold, and its magnitude exceeds one when lr exceeds 2/c, which is outright divergence. The practical translation: high-curvature directions, sharp valleys, tolerate smaller learning rates than flat ones, and a single scalar lr must satisfy the sharpest direction in the surface, which is exactly why ill-conditioned losses train slowly. In multiple dimensions, this is not a hypothetical: real loss surfaces have Hessian eigenvalues spanning orders of magnitude, and the largest one caps the rate for every direction simultaneously.

There is also a subtle interaction with the batch statistics you will meet next. When the gradient is estimated from a noisy minibatch, the effective step includes a random component proportional to the batch's gradient variance; a learning rate that is stable on the full-batch gradient can still diverge on noisy minibatches if the variance times the rate exceeds the curvature's stabilizing pull. This is why batch-size changes often require re-tuning the learning rate in the opposite direction.

def run(lr, w0=0.0, steps=30):
    w = w0
    for _ in range(steps):
        w = w - lr * 2 * (w - 4)
    return w

for lr in [0.01, 0.1, 0.5, 0.9, 1.1]:
    print(lr, round(run(lr), 3))
# 0.01 -> 2.94 (crawling), 0.1 -> 4.0, 0.5 -> 4.0 (edge),
# 0.9 -> oscillates near 4, 1.1 -> grows without bound

Schedules soften the choice. Rather than one constant rate, decay it over training: step decay cuts it by a factor at fixed epochs, cosine annealing sweeps it from a high start to near zero over the run, and warmup ramps it up from tiny values over the first steps so early noisy gradients cannot blow up freshly initialized weights. In practice most production runs use a schedule, and the constant-rate experiment serves as a diagnostic baseline rather than a shipping configuration. Schedules also interact with momentum and adaptive rates, which effectively rescale the step per parameter; a rate that diverges bare may train cleanly once momentum smooths the oscillation, so tune one knob at a time and record what you changed.

Empirical starting points matter in interviews. Classical linear models with normalized features often train stably at 0.01 to 0.1. SGD on neural networks commonly starts near 0.01 or 0.05 with momentum; Adam defaults to 0.001 and tolerates less tuning. These numbers are anchors, not answers: feature scaling changes the curvature, so the same architecture may need a different rate after standardization, and a rate tuned for one batch size may need rescaling for another, a relationship the next topics quantify. The diagnostic loop is always the same: plot the loss, and if it oscillates, halve the rate; if it crawls, multiply it; if it decays smoothly with a shrinking slope, leave the rate alone and spend your effort elsewhere.

The update rule needs a gradient, and where that gradient comes from defines the variant of gradient descent you are running. Batch gradient descent computes it from every training example: sum the per-example gradients over all n samples, average them, and take one step. The step is the exact gradient of the mean loss, so it is deterministic and stable, and it costs a full pass over the data per update.

Formally, the batch gradient is (1/n) times the sum over examples of each example's gradient, which matrix form computes as one product: for squared error it is X^T(Xw - y) scaled, the same expression the normal equation solves in closed form. Every step requires reading all n rows, so a dataset with 10 million examples and a model with a 50-millisecond forward pass takes hours per step, and the total step count to convergence may be in the thousands. That arithmetic is the entire motivation for the sampling-based variants, and it is worth saying plainly: on large data, the full pass, not the arithmetic per example, is the bottleneck.

import numpy as np

def batch_gradient_descent(X, y, lr=0.01, steps=200):
    w = np.zeros(X.shape[1])
    history = []
    for _ in range(steps):
        grad = (2 / len(y)) * X.T @ (X @ w - y)   # full-dataset gradient
        w = w - lr * grad
        history.append(np.mean((X @ w - y) ** 2))
    return w, history

What batch descent buys with its cost is predictability. Each step moves in the true descent direction of the mean loss, so the loss decreases monotonically for a sufficiently small rate, and there is no sampling noise to tune around. On convex losses, the classic result is convergence: with an appropriately decaying rate, the iterates approach the global optimum, and for strongly convex quadratic losses the convergence rate is linear in the number of steps. This is why ordinary least squares, ridge regression, and logistic regression can be trained with full-batch optimization and reported as convex problems with unique solutions, and why a full-batch run is the reference implementation you write first to sanity-check a new model.

What it does not buy is speed, and on non-convex surfaces the deterministic path is not even obviously superior: a smooth path can settle into a saddle point that a noisy path would escape, a fact that motivates the stochastic variants entirely. The naming also matters for interviews: strict textbook gradient descent is batch; the minibatch version that every deep-learning framework implements by default reads the name loosely, computing gradients over small sampled subsets and standing between full-batch stability and single-example noise. That middle variant, and the shuffling discipline it requires, is the next topic.

A hybrid technique worth knowing for interviews is gradient accumulation: when the dataset exceeds GPU memory, you split a logical batch into microbatches, compute each microbatch's gradient, sum them, and only call the optimizer step once you have accumulated the equivalent of the full batch. This gives you the stability of large-batch training with the memory footprint of small-batch training, at the cost of one backward pass per microbatch before a single weight update.