Human Review Workflows

The assistant handles 50,000 conversations a week. Three reviewers are available, and a week of full-time work clears maybe 1,500 outputs. The queue is not a staffing problem; it is a sampling problem. Review everything and the queue grows forever; review nothing and the rare catastrophic output — the wrong refund eligibility, the leaked private detail — ships to customers indefinitely.

Random sampling alone does not fix it. Drawn uniformly, the sample is 95% routine success, and the one catastrophic failure in ten thousand outputs appears in the review queue roughly never. The fix is to spend part of the review budget where the risk is:

import random

def sample_for_review(outputs: list[dict], n: int, risk_threshold: float = 0.7) -> list[dict]:
    risky = [o for o in outputs if o["risk_score"] >= risk_threshold]
    rest = [o for o in outputs if o["risk_score"] < risk_threshold]

    k = n // 2   # half the budget to the riskiest outputs, half to a random cross-section
    picked = random.sample(risky, min(k, len(risky)))
    picked += random.sample(rest, min(n - len(picked), len(rest)))
    return picked

Input: 50,000 weekly outputs, 300 review slots, risk scores from a cheap classifier.

Output: 150 slots to the highest-risk outputs, 150 to a random cross-section of the rest.

The risk score can be simple: low model confidence, a refusal, a high-stakes intent, or a flag from the input filters. The random half is not a concession — it estimates overall quality and catches the failures the risk model never flags. A review program with no random slice has no idea how good the 99% is.

Reviewers are expensive and the task is repetitive, so the labeling interface is a throughput instrument. A case that takes three minutes to grade caps the queue at a few hundred items a week; one that takes seconds clears thousands.

The verdict set should be as small as the decision allows: good / bad / needs-review, with a required one-line reason only for bad. A rubric with seven scales produces careful labels and a queue that never moves; three buttons produce enough signal to find failures and a queue that clears.

Reviewers must not see each other's labels before submitting — visible labels anchor the second reviewer's judgment and destroy the independence that makes adjudication meaningful. The same independence rule applies to model verdicts: a human label recorded next to the judge's score stops being an independent check.

A slice of the review queue — typically 10–20% — is labeled by two reviewers independently. Where they disagree, a third adjudicator breaks the tie. The double-labeled slice is not redundancy; it is the measurement of whether the rubric means the same thing to two people.

def agreement(labels_a: list[str], labels_b: list[str]) -> float:
    agree = sum(a == b for a, b in zip(labels_a, labels_b))
    return agree / len(labels_a)

# two reviewers label the same 200-case slice
# agreement < 0.7 means the rubric is ambiguous — fix the rubric, not the reviewers

Input: 200 cases labeled independently by two reviewers.

Output: agreement 0.68 — the rubric's "correctness" criterion is doing the damage.

Low agreement is a rubric problem until proven otherwise. The usual cause is a criterion written as a feeling ("is the answer helpful?") instead of a check ("does every fact in the answer appear in the retrieved passages?"). Each disagreement cluster gets a rubric revision, and the revised rubric is tested on the same slice before it ships. This is the same calibration discipline as LLM-as-Judge Techniques — the judge is validated against human labels, and here the humans are validated against each other.

The adjudicated label is the gold label. It carries more weight than any single review because it survived a disagreement, and it is the label that enters the dataset.

A review that ends in a ticket is a cost. A review that ends in the dataset is an asset. Every adjudicated correction — the input, the wrong output, the correct label, the rubric clause it violated — is appended to the eval dataset, permanently.

def promote_adjudicated(case: dict, correction: dict, path: str = "evals/support-v1.jsonl") -> None:
    record = {
        "input": case["input"],
        "expected": correction["expected"],   # the adjudicated gold label
        "added": "2026-09-30",
        "source": f"review:{case['review_id']}",
        "rubric_clause": correction["violated_clause"],
    }
    with open(path, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(record) + "\n")

The rubric_clause field is what makes the correction teach. It records not just that this case failed, but which criterion it violated — so the next rubric revision, the next regression suite entry, and the next prompt iteration all inherit the lesson. A correction without the clause is a case; a correction with the clause is a diagnosis.

The full loop, end to end:

production traffic ──► risk + random sampling ──► reviewer labels (seconds per case)
        │                                              │
        │                              10-20% double-labeled ──► agreement measured
        │                                              │
        └──── adjudicated correction ──► eval dataset ──► harness + regression suite
```n
Failure modes that break the loop:

- **Corrections logged and forgotten.** The ticket closes, the label dies with it, and the same failure ships again next month.
- **No random slice.** The risk model's blind spots become the product's blind spots.
- **Adjudication without agreement measurement.** Disagreements are resolved but never counted, so rubric rot goes undetected.
- **Reviewers seeing labels or model verdicts first.** Anchored judgments masquerade as independent ones.

The review loop is where the evaluation system learns from production. Everything upstream — the dataset, the harness, the judge — is downstream of what these labels teach it.