Sampling Parameters and Determinism

A ticket classifier returns billing for "charged twice for invoice 88" on Monday and other on Tuesday for the same string. The prompt never changed. The fix is not in the prompt: sampling ran with temperature left at its default of 1.0, and the runner-up token was close enough to win a coin flip somewhere in the tail.

To see why, follow one token from the model to the wire:

raw logits  -->  penalties  -->  / temperature  -->  top_p filter  -->  sampled token
 (vocabulary scores)      (repeat control)    (reshapes mass)   (cuts the tail)

The model ends every step with a score — a logit — for every token in the vocabulary. Softmax turns those scores into probabilities, and temperature enters between the two: each logit is divided by T before the softmax runs.

p(i) = exp(z_i / T) / sum(exp(z_j / T))

• T < 1 divides the logits down, compressing the gaps and pushing mass toward the winner.
• T = 1 leaves the distribution exactly as the model produced it.
• T > 1 widens the gaps' effect in reverse, flattening toward uniform.

Worked example, three candidate tokens with logits [4.0, 2.0, 1.0]:

T = 1.0   billing ████████████████████ 84%   other ██ 11%    bug █ 4%
T = 0.5   billing ████████████████████████████ 98%  other ▏2%   bug ▏0.2%

The runner-up drops from 11% to 1.8% — and this is per token, hundreds of times per answer. That 11% is where a classification label flips and where a summary wanders off topic; turning temperature down starves it.

reply = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0,        # greedy: always take the argmax
    messages=[{"role": "user", "content": "Ticket: charged twice for invoice 88"}],
)
print(reply.choices[0].message.content)

Input: Ticket: charged twice for invoice 88 with a classify-only system prompt.

Output: billing — and the same output on the next 50 runs.

Temperature 0 is the right default for extraction, classification, and any branch that feeds an if statement. Two caveats: greedy sampling still fails on ambiguous inputs (it picks one label confidently, not the correct one), and temperature 0 is stable rather than bit-forced — provider-side model updates can move the argmax. Reproducibility beyond that is a seed question, taken up at the end.

Temperature reshapes the whole distribution. top_p does something narrower: it discards the tail.

Nucleus sampling sorts tokens by probability and keeps the smallest set whose cumulative probability reaches p. At top_p=0.9, if billing holds 60% and other holds 30%, sampling happens between those two only — every token below the line is impossible for this step, however the probabilities were shaped.

Every knob, and the job each one actually does:

The one-knob rule. Temperature and top_p reshape the same distribution in overlapping ways. Set temperature=1.2 and top_p=0.7 together and a bad output cannot be attributed to either — you have two unmeasured variables in one experiment. Vary one, hold the other at its default, and every A/B run of your prompt tells you something.

reply = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.7,
    top_p=1.0,               # left alone: temperature is today's variable
    frequency_penalty=0.3,   # each repeat costs more than the last
    messages=[{"role": "user",
               "content": "Summarize in two sentences: our migration guide"},
    ],
)
print(reply.choices[0].message.content)

Input: a migration guide that makes every draft say "seamless migration" four times.

Output: two sentences, no phrase repeated — the third occurrence would have been penalized three times as hard as the first.

Penalties apply to tokens the model has already produced in this response, not to your prompt, so they cannot corrupt your instructions. presence_penalty fires once per distinct token — good for topic breadth; frequency_penalty accumulates — good for killing loops. Negative values do the opposite and encourage re-use, which is occasionally what a stylistic echo needs.

The eval suite fails on CI, passes locally, and the diff shows two different labels for one fixed prompt. Non-determinism in tests is not a flake to rerun away — it is a parameter you failed to set.

def run_once() -> str:
    reply = client.chat.completions.create(
        model="gpt-4o-mini-2024-07-18",   # pin a snapshot, not a moving alias
        temperature=0.7,
        seed=42,                          # best-effort reproducible sampling
        messages=[{"role": "user", "content": "Name a metropolis. One word."}],
    )
    return reply.choices[0].message.content

print(run_once())     # Osaka
print(run_once())     # Osaka

Input: the same prompt twice.

Output: Osaka / Osaka — and the same again tomorrow, as long as nothing upstream moves.

A seed requests that the provider reproduce the same sampling choices for identical inputs. What it does not do:

• Survive model updates. A snapshot alias like gpt-4o-mini can be repointed; weights change, probabilities change, and the seed reproduces nothing. Pin the dated snapshot id.
• Force bit-identical outputs. Server-side batching, floating-point reduction order, and replica routing mean identical requests can, rarely, diverge. Where a provider exposes a fingerprint of the running system (OpenAI returns system_fingerprint), record it next to the result — a changed fingerprint invalidates the comparison.
• Fix the prompt side. If your input contains a timestamp, a retrieval hit, or a tool result that varies, the seed is irrelevant: determinism requires identical inputs first.

The discipline this buys, layer by layer:

stable output = pinned model snapshot
              + identical input (no timestamps, no live retrieval)
              + seed set
              + temperature controlled for the task
              + fingerprint recorded

Assertions still belong on structure — the label is one of three enums, the JSON parses, the required keys exist — rather than on exact strings, because exact-string equality is the assertion that fails the day the provider ships a patch. Offline Eval Harness builds suites on that assumption, and Regression Suites covers what to do when a run does move. Determinism governs the content of the answer; whether the answer arrives all at once or in fragments is Streaming Responses and Time-to-First-Token.