RNNs, LSTMs, and GRUs

A recurrent network does one thing: it carries a summary of the past forward while reading the present. The x you feed it is exactly the (batch, T, F) windowed tensor from Sequence Modeling Basics; the recurrence is what stops the history from ending at the window boundary. The step equation fits on one line —

h_t = tanh(W_hh · h_{t-1} + W_xh · x_t + b) — the same weights at every timestep, exactly as a convolution reuses one kernel at every pixel, except here the sharing axis is time.

Unrolled, the model looks like a chain: h_0 (usually zeros) meets x_1 to make h_1, h_1 meets x_2 to make h_2, and so on; the prediction at the end reads h_T. PyTorch hides the loop but keeps its shapes explicit:

import torch, torch.nn as nn

rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True, nonlinearity="tanh")
x = torch.randn(4, 30, 8)              # (batch, timesteps, features)

out, h_n = rnn(x)                      # out: (4, 30, 16) all hidden states
print(h_n.shape)                       # torch.Size([1, 4, 16]) — final state only

# the same step, written by hand:
W_xh, W_hh, b = rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0
h = torch.zeros(4, 16)
for t in range(30):                    # what the framework fuses into one call
    h = torch.tanh(x[:, t] @ W_xh.T + h @ W_hh.T + b)
print(torch.allclose(h, h_n[0], atol=1e-6))   # True

That loop is the whole mechanism — and the whole problem. Predicting from h_1's contribution at position 50 requires backpropagating through 50 chain steps, and each step multiplies by W_hh^T · diag(tanh'). Gradients compose multiplicatively: if the per-step factor averages 0.95, the signal from timestep 1 reaches the loss at 0.95^50 ≈ 0.077 — under 8% — and at 200 steps it is 3.5e-5. That is gradient vanishing, and the practical translation is: the model cannot learn that a token 50 positions ago matters. Flip the factor above 1 and gradients explode instead: 1.03^100 ≈ 19x per passage through the sequence, and your weights jump to NaN by the third epoch. Hence the two standard defenses — gradient clipping, and an activation with bounded derivative:

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # exploding grads

tanh over ReLU in the classic cell is deliberate: tanh' ≤ 1 keeps the product bounded, while unbounded ReLU derivatives compound. Weight initialization matters too — orthogonal W_hh keeps the recurrent product near-isometric so signals neither shrink nor grow for free.

Read the two outputs differently: out holds every timestep's state (use it when each position needs its own prediction — tagging, per-step regression), while h_n is the last state only (use it for sequence-level decisions — sentiment, next-day forecast). With num_layers=2, layer 2 consumes layer 1's outputs as its input sequence and gets its own W matrices, so parameters grow linearly per layer and h_n becomes (2, batch, hidden). Passing your own h_0 lets one sequence's state continue across chunks; forgetting to detach it in a training loop silently backpropagates through chunk boundaries — a classic source of mysterious slowdowns.

Two training details interviewers ask about next. Truncated backpropagation: for a 5,000-step sequence you do not differentiate through all 5,000 — you process in chunks (say 200), backprop within the chunk, and carry h detached to the next. Padded batches: sequences of different lengths either get packed (pack_padded_sequence) so the recurrence never reads pad tokens, or the pad positions get masked out of the loss. What you have now is a model whose entire history squeezes through one 16-float vector, decaying as it travels — the motivation for the cell design in the next chapter, and the baseline Attention Mechanisms will eventually replace.

The LSTM answers the vanishing-gradient problem with a structure change: it adds a cell state c_t — a highway running lengthwise through the chain — and puts gates on-ramps onto it. The step:

f_t = sigmoid(W_f [h_{t-1}, x_t] + b_f)     forget gate:  keep fraction of old memory
i_t = sigmoid(W_i [h_{t-1}, x_t] + b_i)     input gate:   admit fraction of new memory
g_t = tanh    (W_g [h_{t-1}, x_t] + b_g)    candidate:    proposed content in (-1, 1)
o_t = sigmoid(W_o [h_{t-1}, x_t] + b_o)     output gate:  expose fraction of memory

c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t            (elementwise, the memory update)
h_t = o_t ⊙ tanh(c_t)                       (what the next layer actually sees)

Run it with numbers. Suppose c_{t-1} = [1.0, 0.5], the cell decides f_t = [0.9, 0.2] (keep most of slot 1, drop slot 2) and i_t ⊙ g_t = [0.1, 0.3]:

c_t = [0.9*1.0 + 0.1,  0.2*0.5 + 0.3] = [1.0, 0.4]
h_t = [0.6, 0.8] ⊙ tanh([1.0, 0.4]) = [0.6*0.762, 0.8*0.380] = [0.457, 0.304]

The gradient story is the point. Backpropagating the loss from c_T to c_1 gives a product of forget-gate values, ∂c_t/∂c_{t-1} = diag(f_t) — and since 0 ≤ f_t ≤ 1, the product is bounded: when the network learns f ≈ 1, the gradient passes through 50 steps essentially undiminished, the "constant error carousel" from the original LSTM paper. Compare the vanilla RNN's W_hh-product from the previous chapter: no gate, no way to switch the decay off. The model learns when to hold memory (a closing quotation mark or a topic set 200 tokens back) and when to erase it (a sentence boundary) — that is what "gates control what to remember, forget, and emit" means mechanically.

PyTorch usage mirrors nn.RNN, with the cell state returned alongside:

lstm = nn.LSTM(input_size=8, hidden_size=16, batch_first=True)
out, (h_n, c_n) = lstm(x)              # x: (4, 30, 8); c_n: (1, 4, 16)

cell = nn.LSTMCell(8, 16)              # one timestep, weights exposed
# the four gates are fused into one matrix multiply per side:
print(lstm.weight_ih_l0.shape)         # torch.Size([64, 8])  = (4*hidden, input)
print(lstm.weight_hh_l0.shape)         # torch.Size([64, 16]) = (4*hidden, hidden)

The cost is arithmetic: four gates instead of two means 4(H+F)H weights against 2(H+F)H — an LSTM has exactly twice the recurrent matrix parameters of a vanilla RNN (plus two bias vectors per gate in PyTorch, b_ih and b_hh). At F=8, H=16 that is 4*8*16 + 4*16*16 + 8*16 = 512 + 1,024 + 128 = 1,664 parameters — verify it against sum(p.numel() for p in nn.LSTM(8, 16).parameters()) and you have the arithmetic proof.

Keep the two states straight, because interviewers use them as a discriminator. c_t is the raw long-term memory: values ride it unmodified except by the forget gate, which is why the gradient highway lives there. h_t is the filtered read-out — what the next layer and the next step are allowed to see. A cell can be holding crucial information in c while o ≈ 0 keeps h quiet: stored but not currently spoken. The gradient defenses from the vanilla cell apply unchanged — clipping handles explosion through the gates as well, and the bounded sigmoid/tanh derivatives keep the products tame.

Whether that buys anything on your data depends on sequence length and dataset size, and the trimmed-down cell that wins the comparison more often than people expect is the subject of the final chapter.

The GRU (Cho et al., 2014) keeps the LSTM's premise — gated memory — and rebuilds the cell with fewer parts:

z_t = sigmoid(W_z [h_{t-1}, x_t])           update gate:  blend old vs new
r_t = sigmoid(W_r [h_{t-1}, x_t])           reset gate:   how much past to consult
h̃_t = tanh   (W  [x_t, r_t ⊙ h_{t-1}])    candidate from reset history
h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t      one vector, no separate cell state

Two structural differences matter for interviews. First, the cell state disappears — memory lives in h_t itself, so there is no separate highway or output gate deciding what to expose. Second, the update coefficients are tied: (1 - z) and z sum to 1, so the gate cannot amplify — it allocates a fixed budget between preserving the past and admitting the candidate. z_t ≈ 1 means "rewrite"; z_t ≈ 0 means "hold", which is the LSTM's forget-and-input pair compressed into one degree of freedom. The reset gate has its own job: it decides how much of the previous state the candidate is allowed to consult, so after a boundary token (a sentence end, a new topic marker) the model can build the next representation from the current input alone instead of the stale past.

Practical advice carries over from the LSTM unchanged: stack with num_layers, add dropout between stacked layers (PyTorch's dropout= argument applies only between layers, not within the last one), pack padded batches with pack_padded_sequence, and clip gradients regardless of which cell you chose.

Parameter accounting decides cost, and you can compute it exactly for F = 8, H = 256:

def counts(F, H):
    return {
        "rnn": (F*H + H*H + 2*H),
        "gru": (3*F*H + 3*H*H + 6*H),
        "lstm": (4*F*H + 4*H*H + 8*H),
    }

print(counts(8, 256))   # {'rnn': 68096, 'gru': 204288, 'lstm': 272384}
print(counts(8, 256)["gru"] / counts(8, 256)["lstm"])   # 0.75 — GRU is 25% smaller

Decision guidance you can defend in an interview, because there is no universal winner — the original comparison paper found them within noise of each other on most benchmarks:

• Start with the GRU on small or medium datasets: fewer parameters, one less gate to tune, typically faster per epoch, and on short windows (tens of steps) its quality matches.
• Reach for the LSTM when sequences run long or data is large: the separate cell state and output gate give you two independent knobs for "how long to hold" and "how much to show", and the long-range-dependency benchmarks tilt its way.
• Neither, if T is hundreds or thousands. Every step of either cell is sequential, GPU utilization is poor, and both must compress history into one vector. The alternative — let every position attend directly to every other position — is exactly what the next topic builds: Attention Mechanisms.

x = torch.randn(4, 30, 8)
for name, cell in [("GRU", nn.GRU(8, 16, batch_first=True)),
                   ("LSTM", nn.LSTM(8, 16, batch_first=True))]:
    out, h_n = cell(x)
    print(name, out.shape, sum(p.numel() for p in cell.parameters()))
# GRU   (4, 30, 16) 1248   # LSTM  (4, 30, 16) 1664   (H=16 here)

One capability note: both cells accept num_layers= for stacking and bidirectional=True for reading the sequence forward and backward and concatenating states — bidirectionality is free accuracy when the full input is available at once (classification, tagging) and forbidden in streaming, where the future has not arrived. The bigger structural limitation — one vector cannot serve as an addressable memory of a 500-token history — is what attention removes, and it is where this batch's sequence arc ends up.