Model Fallback and Degradation

The pager goes off at 2 AM: the primary model provider is returning 503s. Your on-call checks the status page — a regional outage, ETA unknown. Every feature behind that provider is now a hard error page. Support tickets accumulate. By the time the provider recovers four hours later, you have an incident review, a few angry customers, and a new line in the postmortem: 'single point of failure.'

This is the scenario model fallback exists for. The premise is blunt: any provider you depend on will eventually fail, and the question is not whether your system survives it but what the user sees while it does.

No fallback:
Provider --503--> your service --> user sees error page (100% of traffic)

With fallback chain:
primary --503--> secondary --> user sees answer (maybe lower quality)
both down --> cached/template --> user sees degraded but useful response

The design goal is not perfection — it is graceful degradation. A shorter answer from a smaller model, a cached response, or even a well-written template reply beats an error page, because the user can still accomplish something. The rest of this topic builds the machinery: the fallback chain and health checks, the degradation ladder, and the chaos drills that prove the whole thing works before you need it.

A fallback chain is an ordered list of providers plus the logic that walks it. The logic has two jobs: pick a healthy provider, and react when the one you picked fails.

request --> pick_provider()
               |
               v
         ┌──────────┐  healthy   ┌────────┐
         │ primary  │──────────>│ answer │
         └──────────┘            └────────┘
               │ circuit open (5 failures / 60s)
               v
         ┌──────────┐  healthy   ┌────────┐
         │secondary │──────────>│ answer │
         └──────────┘            └────────┘
               │ also open
               v
         ┌──────────┐            ┌────────┐
         │ degraded │──────────>│cached /│
         │ response │            │template│
         └──────────┘            └────────┘

Health checks come in two flavors. Passive checks infer health from real traffic: consecutive failures and latency percentiles. Active checks probe the provider on a schedule with a lightweight request. Passive checks are free and reflect real user experience; active checks catch a provider that is up but broken for your specific request shape. Production systems use both — passive for the circuit state, active for early warning.

import time

PROVIDERS = ['primary', 'secondary']
FAILURES_TO_OPEN = 5
CIRCUIT_OPEN_FOR = 60  # seconds

failure_counts = {p: 0 for p in PROVIDERS}
last_failure = {p: 0.0 for p in PROVIDERS}

def pick_provider():
    for name in PROVIDERS:
        # a provider re-enters rotation once its cooldown expires — the
        # same half-open idea as a circuit breaker, applied per provider
        if time.monotonic() - last_failure[name] > CIRCUIT_OPEN_FOR:
            return name
    return None  # every provider is down — degrade, do not error

def complete(request):
    provider = pick_provider()
    if provider is None:
        return degraded_response(request)
    try:
        answer = call(provider, request)
        failure_counts[provider] = 0   # any success heals the circuit
        return answer
    except ProviderError:
        failure_counts[provider] += 1
        last_failure[provider] = time.monotonic()
        if failure_counts[provider] == FAILURES_TO_OPEN:
            alert(f'{provider} circuit open — traffic shifting to next tier')
        return complete(request)         # walk to the next provider in the chain

The recursion is the whole design: try the chain in order, heal on any success, and only reach the degraded response when every provider has refused. Note that a success resets the failure count — a provider that fails twice then succeeds once is healthy, and the circuit should say so.

Fallback is not binary. Between 'full answer from the primary model' and 'error page' there is a ladder of degraded modes, and choosing the right rung is a product decision as much as an engineering one.

The ordering principle: degrade the answer before you degrade the availability. A cached answer from yesterday is more useful than an error today, because the user can act on it. A template reply that says 'I cannot answer right now, but here is where to look' preserves trust; an error page destroys it.

Quality differences across the ladder are the reason for alerting. When traffic shifts to the secondary tier, the answers are measurably different — shorter, less nuanced, sometimes wrong in ways the primary never was. If nobody is watching, the backup tier can become the default tier for weeks, and your quality metrics silently erode. Alert on the percentage of traffic served by each tier: anything above a few percent on the secondary is a signal, not a noise floor.

The cached-answers rung deserves a caveat from response caching: only serve cached answers when the cache was warm before the outage. A cold cache during an outage means every request misses, and the 'degraded' tier is slower than the error page it replaced.

Your fallback chain is configured, the code is deployed, and nobody has ever seen it work. That is the default state of failover logic — and the reason it so often fails when finally needed. Untested failover is not failover; it is a hypothesis.

Chaos drills turn the hypothesis into evidence. The practice is simple: in staging, kill the primary provider and assert that the feature still answers.

def chaos_drill():
    with provider_down('primary'):          # staging-only fault injection
        start = time.monotonic()
        answer = complete(sample_request)
        elapsed = time.monotonic() - start
        assert elapsed < 3.0, f'fallback too slow: {elapsed:.2f}s'
        assert answer.source in ('secondary', 'cache', 'template'), \
            f'fallback did not engage: {answer.source}'

Run the drill on every deploy that touches the request path, and after any provider configuration change. The assertions matter as much as the kill: 'still answers' is vague, 'answers within 3 seconds from a non-primary source' is a contract.

The drill usually finds the same three bugs:

• The chain is not actually ordered. A config file lists providers in the wrong order, or an environment variable overrides the list in staging only.
• The degraded response is broken. The template reply references a URL that 404s, or the cache lookup throws instead of missing.
• Healing does not work. The circuit never closes, so once the primary is down it never returns even after recovery — the cooldown logic from the previous chapter is the fix.

Drills also answer the question your incident review will ask: how long does failover take? Measure it in the drill — time from primary failure to first successful secondary answer — and put the number in your runbook. When the real outage comes, 'we drill this monthly and failover takes 400 milliseconds' is the answer that ends the incident review.

Fallback is the last layer of the resilience stack this phase builds: rate limiting and retries keep you from overwhelming providers, and fallback keeps you serving when they are gone. The capstone for this phase, the production RAG assistant, expects all of it.