Sequence Modeling Basics

You have already trained a network on a tabular dataset: shuffle rows, split, fit. Now you are handed 1,000 days of daily sensor readings and asked to predict tomorrow's value — and every instinct from that workflow will betray you twice: once in how you cut the data, once in how you split it. Sequence modeling starts with a single recognition: the row is not the unit of information; the order is. Two rows with identical feature values mean different things depending on when they happened, and a model that cannot see the preceding 30 days is solving a different, harder problem than you intended.

The standard framing converts a series into supervised samples with a sliding window. Given values x_1..x_T, each training example pairs a fixed lookback window with a target horizon steps ahead:

import numpy as np

def make_windows(series, lookback=30, horizon=1):
    n = len(series) - lookback - horizon + 1
    idx = np.arange(n)[:, None] + np.arange(lookback)[None, :]
    X = series[idx]                                  # (n, 30)
    y = series[np.arange(n) + lookback + horizon - 1]  # (n,)
    return X, y

X, y = make_windows(daily_values, lookback=30, horizon=1)
X = X[..., None]                                     # add feature dim -> (n, 30, 1)
# batch convention for sequence models: (batch, timesteps, features)

That (batch, T, F) shape is the contract every model from RNNs, LSTMs, and GRUs onward consumes — T is the axis where order lives, and the rest of this topic's machinery exists to keep that axis meaningful. Real inputs rarely stay one scalar per step: the same window builder stacks features into F = 3 by joining the raw value with day-of-week one-hots and an is_weekend flag, giving X of shape (n, 30, 3) — and any scaler you apply must be fit on the training span alone, then reused for validation and test.

There is a second answer to "what carries older history?" — a model that keeps state between batches: a recurrent net whose hidden state is detached but carried forward so information older than 30 days can still flow. That trade (explicit window versus carried state) is precisely what the next two topics implement, but the data-cutting rules above apply identically to both: order preserved, time-based splits, no normalization fitted across the boundary.

The window length is a modeling decision, not a default you inherit. Too short and the model cannot see structure that exists: a 30-day window captures weekly cycles (every 7th day appears four times) but nothing about an annual one. Too long and you add noise and positions the model must learn to ignore, which on small datasets means overfitting. A quick diagnostic is autocorrelation: if the series correlates strongly at lag 7 but not lag 21, the window must at least cover 7 with room to spare. Windows are lossy — they approximate the full history by discarding everything older than T — and that lossiness is the precise limitation recurrent and attention-based models attack later.

Now the trap that shows up in interviews as a gotcha: overlapping windows must never straddle a random train/validation split. Consecutive windows share 29 of 30 values; put one in training and its neighbor in validation and the model is scored on data it has effectively memorized — validation accuracy soars, real performance does not. The correct cut is chronological: train on days 1-800, validate on 801-900, test on 901-1000, with normalization statistics (mean, std, or a fitted scaler) computed on the training span only. The same principle covers the phase change from vision work — Vision Model Error Analysis taught you to trust sliced metrics over a single accuracy; here the first slice that matters is time.

Suppose the task is now next-token prediction: given "the cat sat on", predict "the". Training has two options for how position 3 gets its context, and the choice shapes everything from gradient quality to whether your model collapses after one mistake.

Option A — teacher forcing — always conditions on the ground-truth previous tokens. During training, the input at every position is the real sequence shifted by one, and the loss is computed at every position simultaneously:

import torch.nn.functional as F

tokens = torch.tensor([[5, 9, 3, 7, 1, 8]])       # one true sequence
logits = model(tokens[:, :-1])                    # input:  everything except last
loss = F.cross_entropy(
    logits.reshape(-1, logits.size(-1)),
    tokens[:, 1:].reshape(-1),                    # target: everything except first
)
# loss = -sum_t log P(y_t | TRUE y_<t, x)

Every position sees correct history, so gradient signal at position 4 does not depend on the model's quality at positions 1-3. Early in training — when predictions are noise — that difference is the difference between a loss curve that falls and one that wanders. It also makes training parallel: one forward pass scores all positions at once, because none of them needs the model's own output as input.

Put numbers on the objective so it stops being a slogan: if the model assigns the true next token probability 0.70, that position contributes -ln(0.70) = 0.357 nats; at 0.05 it contributes -ln(0.05) ≈ 3.00, so one confidently wrong position costs about eight comfortable ones. The training loss is the mean of those terms across positions, and with real contexts each term is directly comparable — the model is graded on what it would do given the truth, every step.

Parallelism is the second payoff. Scoring 500 positions teacher-forced is one batched matmul chain on the sequence; free-running training needs 500 sequential forward passes, each waiting on the previous argmax. On a GPU that is the difference between minutes and hours per epoch, and it is the same structural property that later lets transformers train whole sequences at once.

Option B — free-running training — feeds the model's own previous prediction as the next input, exactly as generation will:

x = prompt
for _ in range(20):
    logits = model(x)[:, -1]
    x = torch.cat([x, logits.argmax(-1, keepdim=True)], dim=1)   # T sequential steps

At step 3 the model conditions on tokens it chose, which mirrors inference — but at epoch 1 those choices are garbage, and the model spends its capacity learning to recover from states it created itself. It trains slowly and often diverges.

The gap between the two is exposure bias: you trained only on clean history, then deploy into free-running generation where one wrong token becomes conditioning context for everything after. Scheduled sampling was proposed to bridge it — with probability p(t) replace the true token with the model's own sample and anneal p — but the empirical picture is that scheduled sampling adds instability and modern language models train with pure teacher forcing, handling self-correction later in training rather than mid-sequences. For interviews, the two-sentence answer is what matters: teacher forcing feeds true previous targets to stabilize and parallelize training; its cost is a train-test mismatch that autoregressive models absorb through scale and later training stages.

You now have the data framing and the training objective. What you cannot yet do is measure honestly — one-step accuracy hides compounding failure — and the next section is the evaluation protocol that exposes it, before the models themselves arrive in RNNs, LSTMs, and GRUs.

A forecasting model reports MAE of 2.3 on your test span and the stand-up goes fine until someone asks: "2.3 compared to what?" The baseline that embarrasses most sequence models is trivial: predict the last observed value (persistence), or for weekly seasonality, the value from the same weekday last week. If your network cannot beat that, it has learned to smooth noise — and random K-fold validation will never have told you so.

Rolling-origin (walk-forward) evaluation is the honest protocol:

def walk_forward(model, series, train_end, horizon=1, steps=50):
    errors = []
    for t in range(train_end, len(series) - horizon, steps):
        window = series[t - 30:t]                       # fixed 30-day lookback
        pred = model.predict(window)                    # trained on series[:t] only
        errors.append(abs(pred - series[t + horizon]))
        # retrain or refit every `steps` origins to simulate real updating
    return float(np.mean(errors)), float(np.sqrt(np.mean(np.square(errors))))

Each origin trains strictly on the past and predicts the future; metrics average over origins instead of one lucky split. Compare the model, the persistence baseline, and last-week-seasonal baseline on the same origins:

Read that table the way an interviewer hopes you will: the MLP beats persistence on one day by a rounding error and loses at the 7-day horizon, so it has learned little beyond "tomorrow resembles today." Horizon is a dimension of evaluation — multi-step forecasts are usually made recursively (predict, append to window, predict again) and errors compound, so a model strong at step 1 can be worthless at step 7. The alternatives — direct models trained per horizon, or a sequence-to-sequence head emitting all 7 at once — trade data efficiency against compounding drift, and you should name that trade when asked.

Two more choices inside walk-forward deserve names before you run one. Refit policy: an expanding window trains on all history at each origin — stable, right when the process is stationary — while a sliding window trains on the last N samples and adapts to concept drift (retail behavior after a redesign) at the cost of data. Refit frequency: retraining at every origin is cleanest but expensive; refit every 30 origins, log the intervening errors, and state the cadence in the report. And never quote a bare mean — across 50 origins report MAE 2.19 ± 0.41, or a bootstrap interval, because a single number hides how lumpy your test span was.

Feature leakage deserves its own warning because it survives a correct split: any column derived from the full series — a rolling mean that includes future rows, a target encoding fitted on all data, a scaler fitted before splitting — imports future information into training windows. Compute every feature causally, from values at or before the row's timestamp, or the walk-forward numbers are fiction.

Two last protocol points close the topic. First, purging: if your label at day t is a 7-day forward return, training and test windows overlap in label time even when split by row — leave a gap of at least the label horizon between them. Second, predictability differs by domain: financial returns sit near a random walk where persistence is close to unbeatable, while daily temperature is highly autocorrelated and easy — reporting "MAE 2.3" means nothing without the baseline row beside it. With framing, training objective, and evaluation locked down, the sequence model itself — hidden state, gates, gradients through time — is next in RNNs, LSTMs, and GRUs.