Reflection and Self-Critique

Fluent, Complete, and Wrong

An agent writes release notes from a merged-PR list. Its draft reads well:

Merged 14 PRs this week: retry budgets, dashboard polish, cache fix.

It also fails three of the four requirements the team specified: no owner named for the cache fix, the schema migration in PR 4471 is missing, and no rollback plan appears anywhere. The prompt asked for all of that.

Nothing went wrong in generation. The model produced fluent text, and fluency is what a single pass optimizes for — no part of generation checks the draft against the requirement list. A one-shot prompt cannot do both jobs well: write freely, then judge what you wrote. So split them into two passes with separate roles:

Critique — read the draft against a checklist and return findings: specific unmet requirements, not general dissatisfaction.
Revision — rewrite the draft addressing exactly those findings.

Keeping the roles separate is what makes the pass measurable. Asked to "improve this," a model rewrites sections that were already correct and can quietly drop a fact that mattered. Asked "here are three findings, fix them," it changes the flagged spans and leaves the rest alone.

The machinery is the one from The Core Agent Loop: critique is the think phase, revision is the act, and the re-graded draft is the observation that decides whether another round happens.

A Critic Armed With a Checklist

The critic's input decides what it can find:

Critic instruction	What comes back	Verifiable?
"Be critical of this draft"	style nits, invented objections, sometimes nothing	no — you cannot tell a real finding from a mood
Four named requirements, each as a yes/no item	findings that map to specific items	yes — count the items still failing

With the checklist fixed, the loop is short:

def critique_and_revise(draft, checklist, ask, grade, max_rounds=2):
    for rnd in range(max_rounds):
        findings = ask.critique(draft, checklist)   # unmet requirements only
        if not findings:                            # clean pass → stop, do not "improve"
            return draft, rnd
        draft = ask.revise(draft, findings)         # fix the flagged spans, keep the rest
    return draft, max_rounds                        # cap reached: accept best effort

Input and output for the release-notes draft:

draft:     "Merged 14 PRs this week: retry budgets, dashboard polish, cache fix."
checklist: owner per change | migration called out | rollback plan | risk level

round 1 findings:
  - owner missing for "cache fix"                (item 1)
  - migration 4471 not mentioned                 (item 2)
  - no rollback plan                             (item 3)
round 1 revision: "Merged 14 PRs. Schema migration 4471 (owner: @dana) ships behind
                   a flag; rollback = redeploy 4470. Also: cache fix (owner: @sam),
                   retry budget tuning (owner: @lee), dashboard polish (owner: @priya)."
round 2 findings: []
status=accepted rounds=2 grade=4/4

Three lines do the real work:

ask.critique(draft, checklist) returns findings as data — item index plus what is missing — so the count is computable. A paragraph of criticism cannot be counted, so it cannot end a loop.
if not findings: return stops the moment the draft is clean. Continuing here would invite the model to polish already-correct sentences and risk breaking them.
max_rounds=2 is a hard ceiling on cost and on drift; the loop never plans to use all of it, only to be allowed to.

The revision call needs three inputs: the current draft, the findings, and the source material the draft summarizes — the PR list, the incident log, whatever ground truth exists. Without the source in that prompt the reviser can only fix structure, so it papers over a missing fact with a plausible-sounding replacement and turns a visible gap into an invisible error. Findings say what is missing; the source says what is true.

Rounds, Caps, and Drift

Reflection has a failure mode that shows up only after round three: drift. Round 1 fixes real findings. Round 2 catches what round 1 missed. Round 3, given nothing to do, starts "fixing" things — it restates a correct owner as a team, converts a specific rollback step into a vague sentence, or drops a number it now considers redundant. Each pass is scored for fluency, so degradation arrives wrapped in smoother prose.

The characteristic ways reflection runs go wrong:

Vague critic. "Be critical" produces style commentary and hallucinated objections. Findings must map to checklist items to be checkable.
Unbounded rounds. Three-plus passes multiply cost per task and walk the draft away from its source facts. Two or three rounds is the practical ceiling; beyond that, quality falls while token spend keeps climbing.
Revision with no findings. A revision pass that runs anyway rewrites correct content — the model has to change something, so it changes what was fine.
Shipped on faith. Reflective output is triple the model calls of a single pass. Without an eval you are paying 3x on the assumption it helps.

That last one is not a feeling you can settle at a desk. Run both variants over the same eval set and compare grades — the harness pattern is covered in Offline Evaluation Harness, with graders like LLM-as-Judge Techniques scoring both drafts against the same checklist. If single-pass wins on your task, reflection is pure cost.

Reflection also spends budget the loop must account for: every round is another model call against the per-task allowance, which is exactly what Stopping Conditions and Budgets, the next topic, puts a number on.

Flashcards

Question

What are the two passes of reflection, and what does each return?

Click to reveal answer

Answer

Critique returns findings — unmet checklist items, empty when clean. Revision returns a draft with only those findings addressed. The empty list ends the loop.

Question

Why give the critic a checklist instead of a general instruction to be critical?

Click to reveal answer

Answer

Checklist findings map to named items, so they are countable and verifiable. 'Be critical' yields style nits and invented objections you cannot check or loop on.

Question

Why cap reflection at two or three rounds?

Click to reveal answer

Answer

Later passes run out of real defects and start editing correct content, so prose gets smoother while facts drift — and every round is another model call.

Question

What is drift in a reflection loop?

Click to reveal answer

Answer

Gradual loss of source facts across revision passes: owners become teams, specific rollback steps become vague sentences, numbers disappear — all while the text reads better.

Question

How do you prove reflection is worth its cost?

Click to reveal answer

Answer

Run reflective and single-pass outputs through the same eval set with the same grader and compare. Reflection stays only if it scores measurably higher.

Revision Notes

Key Takeaways

1. Critique lists unmet requirements as countable findings; revision fixes only those and leaves the rest alone
2. The critic needs a concrete checklist — findings that map to named items are the only kind a loop can act on
3. Stop on the first clean critique and cap total rounds, because extra passes degrade good drafts through drift
4. Reflection triples model calls, so keep it only if an eval shows it beating single-pass output

Interview Tips

• Describe the exit condition first — an empty findings list — then the cap and why it exists
• Give a concrete drift example where round 3 made prose better and facts worse
• Explain why 'improve this draft' fails as a revision instruction and findings-based revision works
• Say how you would measure it: same eval set, same grader, both variants, grades compared

Cheat Sheet