Context Window Budgeting

The chat assistant shipped on Tuesday and returned its first 400 context_length_exceeded on Wednesday, two messages after a user pasted a vendor contract. The deployed model's window is 8,192 tokens; the request carried 9,410. Nothing in the code counts anything: the service concatenates four streams and hopes.

Those four streams are the whole budget:

|<-------------------- context window: 8,192 tokens -------------------->|
[ system prompt ][ conversation history ][ retrieved chunks ][ reserved output ][margin]
      612               variable                variable           1,024        128
      \____________________ what you send ______________________/

• System prompt — instructions, persona, tool and schema definitions. Fixed per request, paid on every call.
• Conversation history — every prior turn you resend. Grows until you evict something.
• Retrieved chunks — passages from search. Grows with k and chunk size.
• Reserved output — the tokens the model will generate. Predicted input plus max_tokens must fit inside the window or the provider rejects the request.

Nothing here amortizes: system prompt and history are re-billed in full on every request, so prompt length is a recurring cost decision rather than a one-time edit.

The arithmetic is subtraction, and it surprises people exactly once:

usable prompt = window - reserved output - safety margin
8,192 - 1,024 - 128 = 7,040 tokens for prompt + history + retrieval

Teams that skip the reserve meet the second failure mode immediately: a request that fills the window completely leaves no room for the answer. The model errors at generation start, or the answer truncates mid-sentence with finish_reason: length while still reading like a finished reply.

Two rules keep the arithmetic honest. The predicted input — system prompt, history, and chunks — plus the max_tokens reserve is checked against the window before dispatch. And the counts come from the model's tokenizer rather than characters, because How Tokenizers Work showed that ratio shifting with language and content.

Assembling Context for the Model decides what goes in; this topic decides how much fits and what leaves when it does not.

Turn the subtraction into a function that also decides what gets evicted. Token counts arrive as integers from a local counter, so the entire budget runs on arithmetic:

def build_request(window, system, turns, chunks, max_output=1024, margin=128):
    budget = window - max_output - margin          # tokens the prompt may occupy
    left = budget - system

    kept_chunks = 0
    for size in reversed(chunks):                  # evict the lowest-ranked chunk first
        if size <= left:
            kept_chunks += 1
            left -= size

    kept_turns = 0
    for size in reversed(turns):                   # evict the oldest turn first
        if size <= left:
            kept_turns += 1
            left -= size

    return {'input': budget - left, 'chunks': kept_chunks,
            'turns': kept_turns, 'free': left}

Input:

print(build_request(
    window=8192,
    system=612,
    turns=[1240, 1180, 1305, 1150, 1260],   # oldest -> newest
    chunks=[318, 302, 295, 341, 287, 310],  # top rank -> lowest rank
))

Output:

{'input': 6180, 'chunks': 6, 'turns': 3, 'free': 860}

Walk it once. The budget is 8,192 - 1,024 - 128 = 7,040; minus the 612-token system prompt leaves 6,428. All six chunks fit (1,853 tokens), leaving 4,575. History arrives oldest first but is consumed newest first: 1,260, then 1,150, then 1,305 fit (3,715 total), and the 1,180-token turn no longer fits — so the two oldest turns, 1,240 and 1,180, are dropped. The request sends 6,180 input tokens and reserves 1,024 for output: 7,204 against a window of 8,192, leaving 988 tokens of slack.

Two lines carry the policy. reversed(...) in both loops states what gets sacrificed first: chunks lose from the tail of the ranked list, history loses its oldest turns, so the freshest dialogue and the best evidence survive. And budget - left is computed after eviction — the number you send is what survived, not what you hoped to send.

The reserve is the line teams delete under deadline pressure. Remove max_output from the subtraction and requests pass validation, then fail at generation, or the answer stops at an arbitrary point. Headroom is not slack; it is the part of the window that makes an answer possible.

Eviction is a policy, and the policy decides which failures users see. Choose it explicitly:

Enforcement happens at three checkpoints:

1. Estimate while assembling. Keep a running token count as you append each component — system, then turns, then chunks — and stop when the budget is spent. Waiting until assembly finishes lets one oversized turn poison the request.
2. Validate before dispatch. input + max_tokens > window is a deterministic check; raise locally with both numbers instead of paying for a 400 the provider will return anyway.
3. Log the survivors. Record input tokens, kept turns, kept chunks, and evictions per request. A steady climb in evictions is the early warning that history or retrieval is growing past budget — Conversation History covers trimming strategies and Token Usage Metering covers the logging layer.

The safety margin exists because your local counter and the provider's may differ slightly — tokenizer version drift, preprocessing of pasted text — and a 128-token cushion keeps that variance from becoming a 400. Treat the margin as an error bar, not as space to reclaim.

Failure modes worth naming:

• Characters instead of tokens — the estimate passes locally and the provider disagrees; the ratios live in How Tokenizers Work.
• Window filled to 100% — no reserve, so generation errors or the answer truncates mid-sentence.
• Evicting newest-first — drops the user's latest question, and the model confidently answers the previous turn.
• Retrying the oversized request unchanged — a wasted round trip that repeats the identical 400.
• Forgetting non-text parts — image tokens, tool schemas, and few-shot examples consume the same budget as prose.

A budget that survives production gives every component a ceiling, subtracts the reserve first, and logs evictions rather than discovering them through a confused user. Once the request fits, the surviving token counts feed Estimating Token Costs directly.