Versioning Prompts in Codebases

At 14:02 an engineer edits the assistant's persona string directly in billing.py to stop it from apologizing twice. The fix ships. At 16:40 a second service — same assistant, prompt defined forty lines away in onboarding.py — starts apologizing twice, because it never received the fix. Rolling back means hunting every copy of that string, and nobody is sure there are only two.

Nothing in the trace metadata records which text was in effect, so reproducing a 15:10 reply means guessing.

prompts/                          <- where the text should live
├── billing-assistant/
│   ├── system.v1.md
│   ├── system.v2.md              <- the 14:02 fix, as a new file
│   └── CHANGELOG.md
└── onboarding/
    ├── system.v1.md
    └── CHANGELOG.md

billing.py    --> load("billing-assistant", "v2")
onboarding.py -> load("billing-assistant", "v2")   # same bytes, one source

A prompt is a production artifact, and a Python string gives it neither a reviewable diff nor a single addressable copy.

from pathlib import Path
import hashlib

def load(name: str, version: str) -> str:
    path = Path("prompts") / name / f"system.v{version}.md"
    return path.read_text(encoding="utf-8").strip()   # trailing newline != new prompt

def prompt_hash(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]

strip() matters more than it looks: a file saved with or without a trailing newline would hash differently, splitting one logical version into two cache entries. The 12-character prefix is short enough for a log line and long enough that collisions are not a practical concern at this scale.

A version number identifies a prompt; the changelog entry is what makes the change reviewable. Prompt prose is hard to judge in a diff — "reworded the refusal line" reads as trivial — so every entry carries the numbers that justified the release:

## billing-assistant/system — v2 (2026-03-11)
- Moved the no-double-apology rule to the end of the prompt
- Dropped the redundant apology sentence
- Eval: triage-v1.jsonl (18 cases) 0.71 -> 0.78
- Regressions: none
- PR #412

What each line answers during review:

Without the eval line a reviewer is reading prose and guessing. With it, the review becomes a claim someone can check: rerun the suite at that commit and the number should reproduce.

Wiring the release into the application stays deliberately boring:

RELEASES = {"production": "v2", "staging": "v3-rc1"}

system_prompt = load("billing-assistant", RELEASES["production"])
trace_meta = {
    "prompt_version": RELEASES["production"],
    "prompt_hash": prompt_hash(system_prompt),
}

Both keys ride on every request trace, so an odd response can be tied back to an exact text — the missing evidence from that Tuesday. When a release starts returning structured payloads instead of prose, JSON Mode Outputs covers what the API guarantees and what your parser still has to do.

Editing a live prompt file in place is pushing straight to production: no review, no eval, no revert point. The pipeline that works for prompts is the one that already works for code.

 branch --> edit prompt file --> run frozen eval suite --> PR review
                                                           |
                              +----------------------------+----------------------------+
                              v                                                         v
                     version bumped,                                          score below bar,
                     changelog appended                                        hold the branch
                              |
                              v
                config pin updated (production: v2 -> v3)
                              |
                              v
                 rollback = repoint the pin to v2

Rollback is a one-line config change rather than archaeology. Both versions still exist on disk, so the old behavior is reproducible without reverting through six files that also moved.

The gate is CI, not memory

A process that relies on someone remembering to run the suite is not a gate. Bind the frozen eval suite to the prompt paths, so any change under prompts/** runs it and reports the score delta against the base branch:

npx evals run --suite triage-v1 --compare main   # fires on every PR touching prompts/**

Then make the two most expensive mistakes testable rather than reviewable:

def test_release_pins_exist():
    for env, version in RELEASES.items():
        path = Path("prompts/billing-assistant") / f"system.v{version}.md"
        assert path.exists(), f"{env} pins a prompt file that is not on disk"

def test_changelog_mentions_current_version():
    log = Path("prompts/billing-assistant/CHANGELOG.md").read_text(encoding="utf-8")
    assert f"v{RELEASES['production']}" in log

The second test catches the actual accident: a prompt file shipped without the entry that records what changed and what the score did.

Caching follows the same idea: key prompt caches and trace records on the hash, never on the name.

cache_key = f"{name}:{prompt_hash(text)}"

A name-based key silently serves v2's cached output after v3 ships, because the name never changed. A hash-based key invalidates itself the moment the text changes, and two services that have converged on identical text share one cache entry even when they loaded it from different paths.

Failure modes of ad hoc prompt changes:

• Silent drift — two services pin different versions of "the same" assistant.
• Unattributable output — no way to tie a response to a text; debugging becomes archaeology.
• Name-keyed cache — stale generations served after the text changed.
• Eval gap — the edit that read as harmless in prose regressed a suite nobody ran.

Versioned files plus scored releases are what let Prompt Iteration Workflow's numbers still mean something three months later, and System Prompt Design is where the text inside those files earns its rules.