Error Recovery in Agent Runs

Last Tuesday an agent got one answer wrong three times in a row: search_kb timed out after 8 seconds, the loop retried the same arguments, timed out, retried, timed out — each retry appending another error to a transcript already at 40k tokens. In a different run, a single PermissionDenied from delete_record was retried eleven times, because the code retried everything.

Neither is recovery. Recovery starts with classification, because the class decides who fixes it:

• Transient — timeouts, 429 rate limits, 502/503. The identical call may succeed shortly; retry with backoff. Your policy fixes it.
• Permanent — 403 permission denied, 404 not found, arguments rejected by validation. The identical call fails identically forever; retrying just turns one error into many. A different tool or a different permission fixes it.
• Malformed action — the model produced arguments that fail the tool schema, so no call happens at all. The fix belongs to the model: return the validation error as the tool result and it corrects itself on the next turn.
• Guard-initiated — your own stop fired, such as a budget or step cap from Stopping Conditions and Budgets. Not a tool failure; retrying it as one re-runs the exact work your guard just ended.

The classification also decides the transport. Every outcome — result or error — must come back from the tool layer as data, never as a raised exception. Raising inside the loop discards every prior turn; a structured payload becomes the tool's message and the conversation survives. That is the difference between a run that degrades and a run that ends.

The retry policy lives in one function whose return value is always a dictionary, so the loop above it can never crash on a tool failure:

import time

RETRYABLE  = {"Timeout", "RateLimited", "ServerError"}
PERMANENT  = {"PermissionDenied", "NotFound", "BadArguments"}

def call_tool(name, args, timeout=8.0, attempts=3):
    """Every path returns a dict: a result, or an error the model can read."""
    last = None
    for attempt in range(attempts):
        try:
            return TOOLS[name](args, timeout=timeout)   # timeout bounds each attempt
        except ToolError as err:
            last = err
            if err.kind in PERMANENT:                   # retrying a 403 is a slower 403
                break
            if attempt < attempts - 1:
                time.sleep(0.5 * 2 ** attempt)          # 0.5s, then 1s — add jitter in prod
        except TimeoutError:
            last = ToolError("Timeout", f"{name} exceeded {timeout}s")
            if attempt == attempts - 1:
                break
            time.sleep(0.5 * 2 ** attempt)
    return {"error": last.kind, "message": str(last),
            "retryable": last.kind in RETRYABLE,
            "attempts": attempts}

A real trace, timeout first and permission second:

search_kb(args={"q": "refund window"})
  attempt 1 → Timeout after 8.0s   (backoff 0.5s)
  attempt 2 → Timeout after 8.0s   (backoff 1.0s)
  attempt 3 → Timeout after 8.0s
  → tool message: {"error": "Timeout", "retryable": true, "attempts": 3}
model next turn → calls kb_cache(query="refund window")   # adapts instead of repeating

delete_record(args={"id": "r_991"})
  attempt 1 → PermissionDenied (403)
  → tool message: {"error": "PermissionDenied", "retryable": false, "attempts": 1}
model next turn → asks for approval instead of retrying

Four lines carry the policy:

1. if err.kind in PERMANENT: break — the fast path. One attempt is already one too many for a 403.
2. 0.5 * 2 ** attempt — exponential backoff, which gives a rate-limited API room to recover instead of being hammered at the original cadence. Jitter it in production or every client retries in lockstep.
3. timeout=timeout per attempt — a tool that hangs is otherwise a run that hangs; no loop guard downstream can fire while a call blocks forever.
4. attempts in the payload — evidence for the model. "Tried three times" tells it to switch tools; a bare error invites the identical call again.

Retries buy time from a recovering service. When the service is not recovering, the run needs an exit that still produces something useful.

Fallback as a declared chain, not improvisation.

primary: search_kb  ──fails 3x──►  fallback: kb_cache (stale, disclosed)
                                 ──unavailable──►  answer with cited gap, or human queue

The chain is configuration, written before the run: primary tool, next-best tool, then a degraded-but-honest output or a human. An improvised fallback — letting the model pick whatever tool it likes after failures — is how an eligibility check becomes a refund. Model-level degradation follows the same shape; see Model Fallback and Degradation.

Checkpoint after every successful step. Persist the message list plus loop state (steps, tokens, scratchpad index) so a crash at step 6 resumes at step 7. Without it, a mid-run failure means paying for steps 1–6 again — and since each round re-sends the transcript, the restart costs more than the work that was lost.

Circuit break on repetition. After N identical failures for the same tool and arguments inside a window, stop retrying entirely and escalate:

if failures[(name, digest(args))] >= 3:      # same call, third failure
    open_circuit(name)                       # block further attempts this run
    escalate(task, last_error, trace)        # queue for a human — see below

The breaker exists because retries are a bet that the service recovers. Three identical failures say it is not recovering, and each further attempt costs tokens, latency, and a transcript slot. Escalation routes to a person through the same pattern as Human Review Workflows.

How runs still go wrong, in the order these appear:

• Retrying permanent errors. A 403 retried nine times is nine billed calls and a longer transcript, with the same outcome.
• No per-attempt timeout. The retry loop is fine but the first call blocks forever, so no guard ever fires.
• Empty or silent error content. Handing the model "" on failure invites it to invent a result; the error payload must say what failed and whether it is retryable.
• Restart from zero. No checkpoints, so a transient crash doubles cost and can produce different answers on the re-run.
• No breaker. Infinite retry against a dead endpoint with nobody paged.

With recovery in place, the remaining question is how tools are declared, versioned, and owned — Designing a Tool Registry, the next topic.