Chain Rule for Backpropagation

A neural network is a function inside a function inside a function: the loss depends on the output, the output on an activation, the activation on a weighted sum, the weighted sum on the weights. To train it you need the derivative of the loss with respect to every weight, and those weights sit five or fifty compositions deep. The chain rule is the tool that turns a derivative through n nested functions into the product of n simple derivatives, each taken one layer at a time.

The single-variable statement: if y = g(u) and u = f(x), then dy/dx = (dy/du) * (du/dx). The composite derivative is the product of the outer derivative (evaluated at the inner function's value) and the inner derivative. That is the entire rule, and everything in backpropagation is bookkeeping around it.

x = 2, w = 3, y_true = 6
u = w * x            = 6        du/dw = x = 2
z = sigmoid(u)       = 0.9975   dz/du = z(1-z) = 0.00249
L = (z - y_true)^2   = 25.0     dL/dz = 2(z - y_true) = -10.005

dL/dw = (dL/dz)(dz/du)(du/dw)
      = (-10.005)(0.00249)(2) = -0.0498

Read that product slowly, because it exposes two facts you will reuse constantly. First, each factor is a local derivative at one node, computable without knowing the rest of the network. Second, the chain can shrink brutally: here the saturated sigmoid contributes a factor of 0.0025, cutting the gradient by 400x in a single layer. Stack ten such layers and the gradient reaching the early weights is multiplied by 400^-10, which is vanishing gradients in its purest form. This is why sigmoid hidden layers lost to ReLU: ReLU's derivative is exactly 1 on the active side, so it multiplies by one instead of shrinking.

Multi-variable chains extend the product to sums. If u depends on w1 and w2, and the loss depends on u, then the partial of the loss with respect to w1 is (dL/du)(du/dw1), and likewise for w2, but if the loss depends on several intermediate paths that converge, the total derivative is the sum over all paths. That path-summing rule is the part students forget, and it is precisely what the backward pass implements when multiple downstream branches consume the same activation: their incoming gradients add up before continuing backward.

Units are the cheapest correctness check for any chain-rule derivation. Treat the derivative as a ratio: dL/dz is loss per unit of z, dz/du is units of z per unit of u, and their product is loss per unit of u, which is exactly what dL/du must mean. If a derived expression's units do not compose, a factor is inverted or missing, and catching that at the whiteboard beats catching it after three days of a model that will not converge. A second useful habit is checking each factor's range: slopes that can only be small (sigmoid's 0.25 ceiling) or only unit (ReLU's active side) immediately bound how gradients behave at depth, before any code runs. The next chapter organizes these facts into the algorithm actually used in every training loop.

Backpropagation is the chain rule organized as an algorithm. Represent every operation of the forward pass as a node in a directed graph: inputs and parameters are leaves, multiplications, additions, activations, and the loss are interior nodes, and the loss is the root. The forward pass computes values along the edges; the backward pass walks the same graph in reverse topological order, carrying one quantity, the gradient of the loss with respect to that node's output, and multiplying by each node's local derivative to produce the gradient with respect to its inputs.

Work a two-layer network by hand. With one example, features x, weights w1 and w2, and sigmoid activations:

FORWARD
h1 = sigmoid(w1 * x)          # hidden activation
h2 = sigmoid(w2 * h1)         # output activation
L  = (h2 - y)^2               # squared error

BACKWARD (reverse order, one local derivative each)
dL/dh2 = 2(h2 - y)                        # loss node
dh2/dh2pre = h2(1 - h2)                  # sigmoid at the output
dh2/dw2 = h1                             # multiplication node
dL/dw2 = dL/dh2 * dh2/dh2pre * dh2/dw2   # chain through to w2
dL/dh1 = dL/dh2 * dh2/dh2pre * w2        # gradient flowing back to h1
# then the same shape again for w1, with sigmoid's h1(1-h1) factor

Notice the two arithmetic patterns. A multiplication node routes its gradient to both parents: the weight's gradient carries the child activation (dL/dw = dL/dh * x), and the input's gradient carries the weight (dL/dx = dL/dh * w), the exact transpose relationship you saw in the linear algebra phase where weight gradients contracted with batch axes. An activation node contributes only its local slope and passes the gradient straight through. The loss node injects the initial gradient, and from there every weight collects its gradient exactly once as the signal passes its edge.

Two algorithmic properties explain backprop's efficiency. First, it computes the gradient for every parameter in roughly the cost of one forward pass: each node's work is a single multiplication, and no derivative is recomputed per parameter. Second, intermediate values from the forward pass (the h1 and h2 above, sigmoid outputs, convolutions' inputs) are reused in the backward pass, which is why frameworks must hold them in memory until backward completes, and why training a network takes several times the memory of inference. Deep networks pay this explicitly: gradient checkpointing trades recomputation for memory by discarding and recomputing selected forward activations.

Reverse topological order is what makes the single pass valid: a node's gradient can only be computed after the gradients of all its consumers have been computed, so the backward walk starts at the loss and fans inward to the leaves, touching each edge exactly once. Any operation that breaks the graph, an in-place modification of a saved tensor, a .detach() on a path that still needs gradients, severs an edge and leaves a leaf with a stale or missing gradient, which is the most common deep-learning bug in production training code. The next chapter shows how frameworks build and execute this graph for you.

PyTorch and TensorFlow build the computation graph during the forward pass and differentiate it on demand, which turns the algorithm above into three lines. Understanding what those lines do, and what state they leave behind, separates engineers who can train models from engineers who can debug training.

In PyTorch, any tensor created from operations on leaf tensors with requires_grad=True records how it was made, storing the operation and its inputs as graph nodes. Calling loss.backward() walks the graph in reverse, computes the gradient of the loss with respect to every leaf carrying requires_grad, and accumulates each result into that leaf's .grad tensor. Accumulation, not assignment, is the first fact to internalize: two consecutive backward calls without optimizer.zero_grad() in between add their gradients, silently doubling the effective learning rate, and the classic symptom is divergence that appears only after you change the batch count or resume from a checkpoint.

import torch

model = torch.nn.Linear(10, 1)
opt = torch.optim.SGD(model.parameters(), lr=0.1)

for x, y in loader:
    opt.zero_grad()                        # clear last step's accumulation
    loss = torch.nn.functional.mse_loss(model(x), y)
    loss.backward()                        # fills model.weight.grad, model.bias.grad
    opt.step()                             # w -= lr * grad for every parameter

The second fact concerns memory. During forward, every intermediate needed by a backward rule is retained: activations for the activation's slope, inputs for the multiplication's gradient. They live until backward runs, and by default the graph itself is freed once backward completes, which is why calling backward twice fails with a graph-freed error unless retain_graph=True keeps it alive (at the cost of the memory you were trying to free). Multiple losses backwarded from one forward, a common pattern in multi-task training, either retain the graph once and backward each scaled loss, or better, sum them and call backward a single time.

The third fact is gradient hygiene. Parameters you intend to freeze get requires_grad_(False) so autograd skips their graph entirely, saving both memory and time; this is how a feature extractor is frozen under a new classification head during transfer learning. Clipping wraps the step (torch.nn.utils.clip_grad_norm_(params, 1.0)), rescaling gradients whose L2 norm exceeds a threshold, which you now read as a norm operation applied to the gradient vector rather than a magic guard. And gradients live on the same device as their parameters, so moving a model to GPU without moving the data produces device mismatch errors at the first backward, not the first forward, because the forward path alone never touches the gradient buffers.

Autograd also explains the deeper pathologies. Vanishing gradients are chains of sub-one local derivatives multiplying to near zero; exploding gradients are chains of large derivatives multiplying without bound, typically from unscaled features feeding recurrent networks. Detached targets, y.detach() in policy-gradient style losses, are deliberate graph cuts that stop gradients from flowing into a network you do not want trained. Every one of these is a statement about edges in the graph, and the graph is exactly the object the chain rule differentiates. With gradient computation settled, the next topic puts those gradients to work: Gradient Descent Fundamentals.