Responding to AI Incidents

The page arrives at 3am: 'support assistant returning garbage.' In a classic web service, that phrase means a bad deploy. In an AI service, it could be five different failures, and the first five minutes determine which runbook you open.

AI-specific failure modes, ranked by how often they page:

The table is the triage. The symptom narrows the failure mode; the failure mode picks the runbook. Skipping triage and guessing is how a 15-minute rollback becomes a 4-hour investigation.

Notice what is missing: 'the model is bad.' Model quality is never the first hypothesis. The model is the same binary it was yesterday; what changed is the prompt, the retrieval, the routing, or the provider. Tracing LLM calls is what lets you check the last deploy in minutes instead of hours.

A provider outage is the most common AI incident and the most rehearsed. The runbook has three moves, in order:

1. Confirm it is the provider, not you. Check the provider status page and your own error rate. If your error rate is flat and the provider is down, the outage is theirs.
2. Flip the fallback flag. Route traffic to a secondary model. This is a feature flag, not a deploy — it takes seconds, not a release cycle.
3. Set the degraded mode. If no fallback model is available, disable the AI feature and serve a static response. A support assistant that says 'I cannot answer right now' is better than one that times out for every user.

def route_model(request, flags):
    if flags.provider_outage and flags.fallback_enabled:
        return FALLBACK_MODEL          # secondary provider, same interface
    if flags.provider_outage:
        return None                     # caller serves the degraded response
    return PRIMARY_MODEL

The fallback model must be tested before the outage, not during it. A fallback that has never served traffic is a hypothesis, not a mitigation. Model fallback patterns and rate limiting with retries are what keep the fallback from cascading — a thundering herd of retries against a recovering provider turns a 5-minute outage into a 30-minute one.

The runbook ends when the provider recovers and the fallback flag flips back. The postmortem starts then — not during the incident.

A quality regression is the incident that looks like a non-incident. Error rates are flat. Latency is flat. The answers are just... worse. Users file tickets with phrases like 'it used to be better,' and nothing pages.

The rollback path must exist before you need it. Two artifacts, pinned and versioned:

• The previous prompt version. Prompts are code. They are deployed, they change behavior, and they regress. Versioning prompts in codebases means the previous version is a known artifact, not a memory.
• The previous model. Model providers update models silently. The model that answered well last week may be a different weights file today.

def rollback_quality(feature: str, registry):
    previous = registry.get_previous(feature)
    registry.deploy(feature, prompt=previous.prompt_version, model=previous.model)
    return previous

# The deploy is a config change: prompt v14 -> v13, model claude-sonnet-4-6 -> claude-sonnet-4-5

The non-obvious requirement is that the rollback is a config change, not a code deploy. A rollback that waits for a CI pipeline is a rollback that takes 20 minutes; one that flips a version pointer takes 20 seconds. The offline eval harness and regression suites are what let you confirm the rollback worked — run the suite against the pinned version before and after the flip.

Detection is the harder half. Quality regressions are caught by online eval metrics — a sampled set of requests scored continuously — not by user tickets. If your only signal is 'users complain,' your detection gap is the incident.

Data exposure is the incident where speed is a legal requirement, not an engineering preference. The runbook is short because the decisions are made before the page:

1. Contain. Revoke the exposed key immediately. A rotated key stops the bleeding; every minute it lives is a minute of exposure.
2. Scope. Search the trace logs for every request that carried the sensitive data. This is why tracing LLM calls records what it records — the trace is the audit trail.
3. Notify. Affected customers and, where applicable, regulators, within the contractual window. The notification is a legal process, not a blog post.

def scope_exposure(trace_store, key_id: str, window_hours: int):
    hits = trace_store.query(
        filter={'attributes.api_key_id': key_id},
        since=timedelta(hours=window_hours),
    )
    return [h.trace_id for h in hits]

The query is only possible because the key ID was a span attribute. Teams that log only 'request succeeded' cannot scope an exposure; teams that log trace context can answer 'who was affected' in minutes.

The postmortem is where AI incidents differ most from classic ones. A blameless postmortem captures the detection gap, not just the root cause:

• Root cause: the key was logged in a span attribute.
• Detection gap: nothing alerted on the key appearing in logs; the exposure was found by a customer.

Fixing the root cause without fixing the detection gap means the next exposure is found the same way. Incident drills exist to close that gap before the real page. The responsible AI foundations — collect only what you need, redact before you log — are the prevention layer that makes the runbook rare.