Prompt Iteration Workflow

A support classifier moved from 71% to 78% accuracy in an afternoon. The change: three reworded rules, one new example, and a temperature tweak, committed together. Two weeks later someone deletes "the redundant sentence" and accuracy falls to 64%.

Nobody can say which of the four edits produced the seven points, so the deletion gets reverted wholesale — along with whatever redundancy it was removing.

  edit bundle --> score up --> which edit? unknown --> later cleanup --> score down
                                                                      (revert all)

The measurement is what broke, not the prompt. A score attached to a bundle of changes carries no information about any individual change, and every later edit inherits a baseline nobody trusts.

Two rules restore the loop:

1. Freeze the eval set. A fixed list of inputs — good cases, bad cases, edge cases — scored the same way every time. If the inputs move, the score compares to nothing.
2. Change one variable at a time. Reword one rule, measure. Add one example, measure. Move the temperature, measure. Each score stays attributable to the edit that produced it.

The habit that makes rule 2 stick: write down your expectation before you run it. "Removing the duplicate sentence should not change accuracy; if it does, the model was leaning on that sentence." Predictions turn a scoreboard into evidence.

The loop is small enough to run in a terminal:

 1. FREEZE    fixed cases in evals/triage-v1.jsonl
 2. CHANGE    exactly one edit to the prompt
 3. SCORE     run every case, compare against expected
 4. RECORD    keep the scores, and keep the failures

import json
from openai import OpenAI

client = OpenAI()
CASES = json.load(open("evals/triage-v1.jsonl"))   # never edited mid-experiment

def run(prompt: str, text: str) -> str:
    reply = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{prompt}\n\n{text}"}],
        temperature=0,
    )
    return reply.choices[0].message.content.strip()

def score(prompt: str) -> tuple[int, list[dict]]:
    missed = []
    for case in CASES:
        got = run(prompt, case["input"])
        if got != case["expected"]:
            missed.append({**case, "got": got})
    return len(CASES) - len(missed), missed

baseline = "Classify the ticket as billing, shipping, or account. One word only."
edited = baseline.replace(
    "One word only.",                      # the one edit, visible in the diff
    "One word only, never a full sentence.",
)

before, failures = score(baseline)
after, regressions = score(edited)
print(f"{before}/{len(CASES)} -> {after}/{len(CASES)}")
for r in regressions:
    print("REGRESSION:", r["input"], "->", r["got"])

str.replace makes the single-variable rule literal: the difference between the two prompt strings is one phrase, and a reviewer can confirm that from the source. The failure list prints per run because 16/18 and 15/19 are nearly the same number — the inputs that broke are the artifact, not the fraction.

Each stage leaves something on disk:

Exact-match scoring stops being enough the moment output is free text — a summary can be correct in five different wordings. At that point scoring becomes a component with its own prompt, which is LLM-as-Judge; building the case list itself is Building Eval Datasets.

A suite that only holds passing cases is a snapshot. A suite that grows every time production surprises you is a regression corpus. The rule: every failure found in a run, a log review, or a bug report becomes a case with a human-written expected answer, then stays in the file forever.

def promote(failure: dict, path: str = "evals/triage-v1.jsonl") -> None:
    case = {
        "input": failure["input"],
        "expected": failure["expected"],   # written by a person, once
        "added": "2026-03-11",
        "source": "INC-4471",
    }
    with open(path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(case) + "\n")  # JSONL: append-only, diffable line by line

Scoring free text needs a judge, and the judge needs constraints of its own:

Grade the candidate answer against the reference.
Reply with exactly one word: PASS or FAIL.
Judge factual agreement only; ignore wording differences that do not change meaning.

Fixed judge prompt, fixed judge model, temperature 0 — change any of the three and yesterday's scores stop meaning what they meant. The judge also needs human calibration: read about 10% of its verdicts each cycle, and treat a disagreement rate above roughly one in ten as a bug in the judge rather than a win for the prompt.

Failure modes of the loop itself:

• Moving goalposts — adding cases mid-experiment shifts the score without any prompt changing. Add cases between experiments.
• Judge rewards verbosity — a longer, hedged answer can pass a vague judge. The one-word verdict format removes most of that.
• Overfitting the suite — a prompt tuned against 18 known cases fails case 19. Hold back a small set you never tune against.
• Bare numbers — a percentage in a spreadsheet is not a record; store the failing inputs beside it.

What you are building — versioned prompt, dated scores, retained failures — is exactly what Versioning Prompts in Codebases needs to make a release reviewable, and it is the measurement harness Chain-of-Thought Prompting should be scored against before you pay for traces.