LLM-as-Judge Techniques

A prompt change ships on Monday. By Friday the team has read maybe thirty outputs, all of them fine, and a quiet suspicion that something else broke. Human review is the ground truth of AI evaluation, and it does not scale: three reviewers reading a thousand outputs a week is a full-time job, and the week after the next prompt change the queue is longer.

LLM-as-judge attacks the queue directly. A strong model can grade thousands of outputs against a written rubric, consistently, at the cost of API tokens instead of human hours. The catch is that an unconstrained judge — "rate this answer 1 to 5" — is worse than useless: its scores drift with wording, length, and the judge's own preferences, and a drifting score is noise wearing a metric's clothing.

The techniques that make a judge trustworthy are all constraints on its freedom:

• Rubrics with anchors pin each score level to a concrete example, so "good" means the same thing on every run.
• Pairwise comparison asks the judge to choose between two outputs, a judgment models make far more reliably than absolute scoring.
• Position swapping detects the bias toward whichever answer appears first.
• Human calibration measures agreement with people before any aggregate score is believed.

The dataset these judges run over is the one from Building Eval Datasets; the judge is the scoring layer on top of it.

A rubric converts "grade this answer" into a checklist, and anchors convert each checklist item into something the judge can pattern-match against. The difference between the two prompts is the difference between a score that reproduces and one that wanders.

JUDGE_PROMPT = """You are grading a customer-support answer.

Score each criterion 0 or 1:
1. Correctness: every fact in the answer is true.
2. Completeness: the answer resolves the question fully.
3. Safety: no harmful, illegal, or private information.

Anchors:
- Correctness=1: "Refunds post within 3-5 business days." (matches policy)
- Correctness=0: "Refunds are instant." (contradicts policy)
- Completeness=1: states the amount, the method, and the timeline
- Completeness=0: answers only part of the question

Answer to grade:
{answer}

Return JSON: {"correctness": 0, "completeness": 1, "safety": 1, "reason": "..."}"""

Input: an answer claiming refunds are instant, against a policy that says 3-5 days.

Output: {"correctness": 0, "completeness": 1, "safety": 1, "reason": "refund timeline contradicts policy"}

Three properties do the work:

• Binary criteria. "Correctness: 0 or 1" is a far more consistent judgment than "quality: 1 to 5." Absolute scales ask the judge to calibrate a number against an internal feeling; binary checks ask it to verify a claim.
• Anchors are examples, not definitions. The anchor for Correctness=0 shows the judge what a violation looks like in your domain. Without anchors, the judge imports its own standard of "correct," which is not your policy.
• The reason field is for humans. A one-line justification lets a reviewer audit a sample of verdicts cheaply — the calibration step in the next chapter depends on it.

Judge prompts are versioned like code. Change the rubric and yesterday's scores stop meaning what they meant, so the prompt text, the model, and the temperature are recorded alongside every run.

Asking a judge to score a single output on an absolute scale is asking it to hold a standard in its head. Asking it to choose the better of two outputs is asking it to make a relative judgment, and relative judgments are measurably more consistent. For open-ended generation — summaries, drafts, explanations — pairwise comparison is the default.

import json, random

def judge_pair(prompt: str, answer_a: str, answer_b: str, ask) -> str:
    swap = random.random() < 0.5
    first, second = (answer_b, answer_a) if swap else (answer_a, answer_b)
    resp = ask(
        f"Which answer better fulfills the request? Judge correctness and completeness.\n\n"
        f"Request: {prompt}\n\nAnswer 1: {first}\n\nAnswer 2: {second}\n\n"
        f'Return JSON: {{"winner": "1 or 2", "reason": "..."}}'
    )
    winner = json.loads(resp)["winner"]
    if swap:
        winner = {"1": "2", "2": "1"}[winner]   # undo the swap before reporting
    return "A" if winner == "1" else "B"

Input: the same pair judged ten times with random swaps.

Output: A wins 7, B wins 3 — and the raw "Answer 1" count is 5, not 7, because the swaps were undone.

The swap line is the bias detector. Judges exhibit position bias: all else equal, they favor whichever answer appears first. If the raw "Answer 1" count across many pairs lands far from 50%, the judge is reading order, not quality. Swapping randomly and undoing the mapping removes the bias from the verdict; the raw count is the diagnostic that tells you it was there.

Pairwise judging has a cost: comparing N outputs needs O(N²) judgments for a full ranking. In practice teams judge a candidate against the current champion only — O(N) judgments — and accept that the ranking is only pairwise-consistent, not globally optimal.

A judge that agrees with itself is not necessarily right. Before any aggregate score is trusted, the judge's verdicts are measured against human labels on a sample — and the disagreement rate is treated as a bug in the judge, not a win for the prompt.

The protocol is deliberately boring:

1. Take 100–200 cases from the eval dataset that humans have already labeled.
2. Run the judge over the same cases with the same outputs.
3. Compute agreement: the share of verdicts where judge and human match.
4. Read the disagreements. Each one is either a judge error or a rubric ambiguity — fix the rubric, re-run.

Failure modes that survive a careless calibration:

• Self-preference bias. Judges favor outputs from their own provider or model family. If the judge is the same family as the system under test, its scores are inflated by construction — use a different model for the judge.
• Verbosity bias. Longer, hedged answers read as more thorough. Binary criteria and anchors push back against it, but a sample of long-answer disagreements should be read by a human every cycle.
• Calibrating on the tuning set. Agreement measured on cases the prompt was tuned against is optimistic. Calibrate on a fresh sample or a slice of the holdout.

The judge is a scorer, not an oracle. Its verdicts feed the offline harness, which is where these scores become a gate on shipping.