Derivatives and Gradients

Training a model means minimizing a loss, and minimizing any function means knowing precisely how it changes when you nudge its inputs. That knowledge is exactly what a derivative provides. The derivative of f with respect to x answers one question: if x moves a tiny amount, how much does f move, and in which direction? Geometrically it is the slope of the tangent line at that point; numerically it is the ratio of a small change in f to the small change in x that caused it.

The ML case is concrete. Suppose your model makes one prediction, y_hat = w * x for a single training example with x = 2 and true label y = 6, and you use squared error as the loss: L(w) = (w*x - y)^2 = (2w - 6)^2. Expand the square and differentiate the resulting polynomial term by term:

L(w) = (2w - 6)^2 = 4w^2 - 24w + 36
dL/dw = 8w - 24

At w = 3 the prediction is exactly right (23 = 6), and the derivative is 83 - 24 = 0: a flat tangent, a minimum, no update needed. At w = 1 the derivative is -16: increasing w reduces the loss, steeply. At w = 5 it is +16: increasing w raises the loss, so the update must move the other way. Every gradient descent step you will ever run reads the derivative at the current point and steps opposite its sign; the magnitude tells you how far that step can afford to go.

import numpy as np

def loss(w):        return (2 * w - 6) ** 2
def dloss(w):       return 8 * w - 24

for w in [1.0, 3.0, 5.0]:
    eps = 1e-5
    numeric = (loss(w + eps) - loss(w - eps)) / (2 * eps)
    print(w, dloss(w), round(numeric, 4))   # analytic matches numerical

The numerical check in that snippet is more than a pedagogical trick; it is how you verify any hand-derived or auto-differentiated gradient. Move the input by a tiny epsilon on both sides, divide the loss difference by 2*epsilon, and compare with the formula. Frameworks ship this as gradcheck, and running it after implementing a custom layer catches sign and scaling bugs before they corrupt training silently. The epsilon cannot be arbitrary either: too large and truncation error dominates, too small and floating-point rounding swamps the difference, which is why the symmetric centered form is preferred over the one-sided one.

Derivatives of the losses you will meet follow simple patterns worth memorizing. Squared error gives a derivative proportional to the error itself, so large misses dominate training and a single mislabeled outlier can steer a whole batch. Absolute error gives a constant-magnitude derivative (plus/minus one), which resists outliers but stalls exactly at zero error. Cross-entropy with a sigmoid produces the remarkably compact (prediction minus label), a fact you will derive with the chain rule and then see reused in every classification training loop.

Verifying Gradients Numerically

A practical way to validate any hand-derived gradient is the centered difference approximation. For a function L(w), the numerical derivative at w is (L(w + \u03b5) - L(w - \u03b5)) / (2\u03b5) for a small \u03b5 (e.g., 1e-5). When this matches the analytic derivative dL/dw within a tight tolerance, you have high confidence the gradient is correct. Frameworks like PyTorch provide orch.autograd.gradcheck() that automates this verification, making it a standard practice before deploying custom layers to production.

Choosing Learning Rates Based on Gradient Magnitude
The magnitude of the gradient directly informs the safe step size for gradient descent. If the gradient is large, a large learning rate may cause the parameters to overshoot the minimum, leading to divergence or oscillation. If the gradient is tiny, progress will be painfully slow with a fixed learning rate. This is why many frameworks offer learning rate schedules or adaptive methods like Adam that adjust the step size per parameter based on historical gradient magnitudes, ensuring stable convergence across diverse loss landscapes.

A model does not have one weight; it has thousands, and the loss depends on all of them at once. The gradient extends the single derivative to this setting. The partial derivative of L with respect to wi holds every other weight fixed and asks how L changes as wi alone moves. Collecting all of those partials into a vector gives the gradient, written with the nabla symbol, and it has one crucial property: it points in the direction of steepest increase of L at that point. To descend, you step against it.

Work a two-weight example. Let L(w1, w2) = w1^2 + 3w2^2. The partials are 2w1 and 6*w2, so the gradient at (1, 1) is (2, 6). The w2 direction is three times steeper, so a step of equal length against the gradient reduces w2's contribution far more. This anisotropy, unequal steepness across directions, is the single most important geometric feature of real loss surfaces, and it is why the optimization chapters ahead introduce momentum and adaptive learning rates.

import numpy as np

def L(w):      return w[0] ** 2 + 3 * w[1] ** 2
def grad(w):   return np.array([2 * w[0], 6 * w[1]])

w = np.array([1.0, 1.0])
eta = 0.1
for step in range(12):
    w = w - eta * grad(w)         # step opposite the gradient
print(w, L(w))                    # near zero, loss near zero

Two properties of the gradient are interview staples. First, it is the direction of steepest ascent locally, which follows from the dot product: among all unit step directions, the one maximizing the increase in L aligns with the gradient, and the rate of increase along direction d is the dot product of the gradient with d. Second, the gradient is local: it describes the surface at the current point only, and a large step can cross a valley and land somewhere the local picture no longer applies, which is exactly how too-large learning rates diverge. Together they explain why the update rule uses a small learning rate: the gradient's promise holds only in a neighborhood, and the step size decides how much of that promise you can collect.

Autograd is why you rarely compute these by hand. Frameworks record the operations of the forward pass into a computation graph, then apply the chain rule backward over it to fill a gradient for every leaf tensor. loss.backward() in PyTorch populates param.grad for every parameter; optimizer.step() consumes it. The practical consequence for debugging: gradients accumulate until you zero them, and a parameter whose .grad stays None or zero after backward usually sits on a disconnected part of the graph (detached tensor, in-place operation breaking the record), a diagnosis you can perform only if you understand what the gradient vector represents. Working one two-weight gradient by hand first also makes the automatic version trustworthy: you can predict what .grad should contain before you run backward, which turns autograd from a black box into a cross-checkable tool. The mechanics of that backward sweep are the next topic: the chain rule.

Plotting the loss against a parameter, against training steps, or against data size turns calculus into a diagnostic instrument. The shapes you see map directly onto derivative facts, and experienced engineers read them the way a mechanic listens to an engine.

Start with the one-parameter view: loss versus w for a convex squared-error problem is a parabola, the simplest curved surface there is. At the bottom the derivative is zero, which is the definition of a minimum. Far up the sides the derivative is large, promising big reductions per step. Near the bottom the derivative shrinks, so progress slows, which is why a constant learning rate takes ever smaller effective steps in parameter terms while the loss plateaus. If the curve is a flat line everywhere, the derivative is zero across the range, meaning the features carry no signal or the activation saturated; if it is steep and noisy, the batch used to estimate the derivative is too small or the features are unscaled.

import numpy as np
import matplotlib.pyplot as plt

w = np.linspace(-1.0, 4.0, 200)
L = (2 * w - 6) ** 2
plt.plot(w, L)                       # parabola, minimum at w = 3
plt.plot(w, 8 * w - 24)              # tangent slope: the derivative line
plt.axhline(0, color='gray')

# training curve: loss vs step, the production workhorse
for step in range(200):
    loss_val = train_one_step(model)  # plot this against step

The training curve, loss against optimization step, carries its own vocabulary. A steadily decaying curve with a shrinking slope is healthy: the gradient magnitude is falling as the surface flattens near a minimum. A curve that oscillates up and down means steps overshoot the valley: the gradient reverses sign every step because the learning rate exceeds what the curvature allows, visible in the derivative analysis as a step larger than twice the distance to the minimum. A curve that plateaus early at a high loss indicates a saturated activation or a dying ReLU, where the derivative is zero for large regions and the gradient carries no information whatsoever. A training loss that falls while validation loss rises is overfitting, a data issue no learning-rate change will fix.

Two extensions complete the picture. Loss surfaces in many dimensions are not parabolas; they contain saddle points, where the gradient is zero but the point is not a minimum, distinguishable only by curvature (the eigenvalues of the Hessian you met in the linear algebra phase). And the loss-versus-data-size curve, learning curve, tells you whether collecting more data would help: if validation loss is still falling as training set size grows, more data buys accuracy; if it has flattened, the model family or features are the binding constraint. Both appear regularly in interviews as questions about what you would try next, and both are derivative statements about the shape of a surface, which is the habit this phase builds: read the shape, name the derivative fact, choose the intervention. The mechanism that computes these derivatives through dozens of nested functions is the chain rule, covered next.