Managing Conversation History

A bug report lands: the assistant answered with a flight date the user never mentioned in this session. You open the service logs and find the latest user string and nothing else — the full messages array that was actually sent was never written anywhere, so the bug cannot be reproduced.

A turn is the unit of that record. Persist one row per message rather than one row per exchange, because replay has to rebuild the array exactly:

from dataclasses import dataclass, field
import time, tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")

@dataclass
class Turn:
    conversation_id: str
    seq: int                       # monotonic within a conversation
    role: str                      # system | user | assistant
    content: str
    created_at: float = field(default_factory=time.time)
    tokens: int = 0                # filled at write time

    def __post_init__(self):
        self.tokens = len(enc.encode(self.content))

def append_turn(conversation_id, turns, role, content):
    turns.append(Turn(conversation_id, len(turns), role, content))

Input: append_turn("c-9f2", [], "user", "Move my standup from the 14th to the 21st")

Output: Turn(conversation_id='c-9f2', seq=0, role='user', content='Move my standup from the 14th to the 21st', created_at=1750000000.31, tokens=13)

Each column earns its place. seq restores order when two workers write concurrently. created_at answers "what did the model see at 10:04?" without parsing log timestamps. tokens is computed once at write time, so trimming later counts integers instead of re-encoding the whole log. Role and content stay in separate fields because the resend step needs the array shape — Chat Messages and Roles covers that contract.

Storing the whole request body as one blob is the tempting shortcut: one insert per exchange instead of one per message. It defeats per-turn token accounting, partial replay, and the ownership filter from the next chapter, all of which need to address a single turn.

Replay is then a query, not a feature: SELECT seq, role, content FROM turns WHERE conversation_id = %s ORDER BY seq returns the array in the only order that matters — the order it was sent.

History grows monotonically and the context window does not, so every request needs a declared rule for what gets sent. "Send everything" works until turn 40, and its failure mode is a provider 400 rather than a visible bug in your code.

Four retention policies, each trading one failure for another:

A token-budgeted window is the sliding policy with an honest counter:

def window(history, budget=1500):
    system = [t for t in history if t.role == "system"]   # pinned, never enters the loop
    body   = [t for t in history if t.role != "system"]

    used, kept = sum(t.tokens for t in system), []
    for t in reversed(body):                              # newest first
        if used + t.tokens > budget:
            break
        kept.insert(0, t)
        used += t.tokens

    while kept and kept[0].role != "user":                # strict endpoints reject an assistant-first array
        kept.pop(0)
    return system + kept

Input: 6 turns, system prompt 40 tokens, turn counts [310, 295, 402, 388, 517, 64], budget 1500.

Output: system prompt plus the 4 newest turns (seq 2–5): 1,411 tokens sent, 89 left under budget.

Walking it: the loop pulls seq 5 (64), 4 (517), 3 (388) and 2 (402) — 1,371 body tokens, 1,411 with the 40-token system prompt. The next candidate, 295, would reach 1,706 and blow the 1,500 budget, so seq 0 and 1 are dropped. Two lines carry the policy. reversed(body) declares that the oldest turns are sacrificed first, which keeps the current question alive; and the system entry is filtered out before the loop, so standing instructions survive every trim. The final while removes an orphaned assistant turn so strict endpoints such as Anthropic's do not answer with 400: roles must alternate.

How much history the budget may claim, and where the reserve comes from, is Context Window Budgeting. When dropping is the wrong answer — the session holds a decision the user made at turn 3 — the escape hatch is Summarizing Long Conversations.

A single-user demo stores history under one key. Production stores thousands of conversations under keys built from two identifiers, and its interesting failures are authorization failures, not storage failures.

The key is user_id : conversation_id, and both parts belong inside the query rather than in a handler that a future endpoint might skip. conversation_id alone lets a client holding a guessed or leaked id read someone else's turns; user_id alone collapses a user's unrelated conversations into one stream.

def load_history(pool, user_id, conversation_id, limit=200):
    with pool.cursor() as cur:
        cur.execute(
            """SELECT seq, role, content, tokens FROM turns
               WHERE user_id = %s AND conversation_id = %s
               ORDER BY seq DESC LIMIT %s""",
            (user_id, conversation_id, limit),
        )
        rows = cur.fetchall()      # both predicates are in WHERE: no check to forget
    return list(reversed(rows))

Input: load_history(pool, "u-17", "c-9f2") where conversation c-9f2 belongs to u-44.

Output: [] — the row exists, but the user_id predicate excludes it, so the caller sees an empty conversation instead of another user's turns.

Failure modes to design against:

• Enumerable ids — sequential conversation_id values are guessable; use unguessable ids and filter by owner, because either one alone is a weak wall.
• Check in the handler, not the query — a new route reusing the loader without re-adding the predicate inherits the hole silently.
• Two writers, one conversation — two tabs appending to the same key interleave seq values and corrupt replay; Session Persistence Patterns covers versioned writes for exactly this case.
• Unbounded reads — without LIMIT, a 5,000-turn conversation ships its entire log on every request; the replay window is a cost decision as much as a correctness one.
• No purge policy — rows outlive the session. Pick a retention window deliberately and enforce it with a scheduled delete, not with hope.

Where the log itself lives is a storage decision covered in Choosing a Vector Store for turn retrieval setups. What gets sent back to the model on the next turn is the trimming decision above; whether the dropped turns are compressed instead is Summarizing Long Conversations.