AI Deployment Checklist

Most AI outages trace back to two mundane causes: a key that leaked into a repository, or a provider quota that evaporated under real traffic. Both are checkable in an afternoon, and both are invisible until they fail.

Secrets first. API keys live in a secret manager and are injected at runtime — never in code, never in client bundles, never in environment files committed to the repo. Rotation is a documented procedure, not a hope: generate the new key, verify it, swap it, revoke the old one, and test the whole path before you need it under pressure.

import os

def load_config() -> dict:
    api_key = os.environ["LLM_API_KEY"]  # fails fast if the secret is missing
    return {
        "api_key": api_key,
        "model": os.environ.get("LLM_MODEL", "provider-default"),
        "max_retries": int(os.environ.get("LLM_MAX_RETRIES", "3")),
    }

os.environ["LLM_API_KEY"] raises immediately when the key is absent, so a misconfigured deploy fails at startup instead of on the first user request. Quotas second: confirm your provider limit is at least 3x your expected peak requests per second, and confirm the burst ceiling covers a retry storm. A quota exhausted at 10x traffic is a capacity problem, not a code problem — see Rate Limiting and Retry Policy for client-side protection.

The checklist item that catches both: run the pre-flight check as part of CI, not as a manual step before launch. A script that asserts the key loads from the environment, the model name is pinned, and the quota header exceeds your peak turns a 3 a.m. page into a failed build. Readiness you can automate is readiness you can trust.

Load testing with "hello" tells you nothing. Prompt size drives token count, latency, and cost, so the test corpus must be sampled from real traffic — including the long, messy prompts users actually send.

import asyncio, time

async def load_test(request_fn, prompts: list[str], concurrency: int = 50) -> dict:
    sem = asyncio.Semaphore(concurrency)
    latencies = []

    async def one(prompt: str):
        async with sem:
            start = time.perf_counter()
            await request_fn(prompt)
            latencies.append(time.perf_counter() - start)

    await asyncio.gather(*(one(p) for p in prompts))
    latencies.sort()
    return {
        "requests": len(latencies),
        "p50_ms": latencies[len(latencies) // 2] * 1000,
        "p95_ms": latencies[int(len(latencies) * 0.95)] * 1000,
    }

Input: 1,000 prompts sampled from production traffic, run at 50 concurrent. Output: p50 420ms, p95 1,900ms, zero errors. Read the p95, not the mean — the mean hides the tail where your users live. If p95 exceeds your latency budget, the fix is usually a smaller model tier, a cache, or a reranker budget, not more hardware. For where the time actually goes, see Profiling End-to-End Latency.

Concurrency deserves a deliberate number, not a guess. Start at your expected peak concurrent requests and hold it for ten minutes — a burst test that passes for thirty seconds tells you nothing about connection pools, queue depth, or whether your rate limiter engages before the provider's does. If errors appear only at high concurrency, the fix is client-side throttling, not a bigger quota.

You should learn about your own outage from your dashboard, not from a customer. Three metrics carry an AI feature: error rate, latency, and spend. Each gets a dashboard and an alert threshold before launch day.

def emit_metrics(request: dict, latency_ms: float, cost_usd: float) -> None:
    metrics.gauge("llm.latency_ms", latency_ms, tags={"model": request["model"]})
    metrics.gauge("llm.cost_usd", cost_usd, tags={"feature": request["feature"]})
    metrics.increment("llm.requests", tags={"model": request["model"], "status": request["status"]})

Tagging cost by feature is what makes a blowout diagnosable: a runaway loop in one feature shows up as a spike in one tag, not as a mysterious rise in a fleet-wide total. See Metering Token Usage and Tracing LLM Calls for the full trace design.

Alert fatigue is the silent killer of observability: ten alerts that fire weekly train the team to ignore the eleventh. Every threshold above should page a human only when it needs a human — error rate and spend get pages, latency gets a ticket, and anything that recovers on its own gets a log line. The test is simple: if an alert has never been worth waking someone for, it is not an alert.

A bad prompt release is a matter of when, not if. The question is whether you can reverse it in minutes or in hours. Prompts and models are versioned artifacts — stored, reviewed, and promoted through the same pipeline as code — so "rollback" means flipping a flag, not redeploying.

Rollback steps, in order: (1) flip the feature flag to the previous prompt version; (2) watch the error rate and faithfulness dashboards for 10 minutes; (3) if the regression is in the model, pin the previous model version; (4) write the postmortem while the details are fresh.

The runbook documents this before launch, not during the incident: what the dashboards are called, where the flags live, who has permission to flip them, and what "done" looks like. A runbook written at 2 a.m. during an outage is a runbook with mistakes. For versioning practice, see Versioning Prompts in Codebases; for model fallback design, see Model Fallback and Degradation.

Test the rollback in staging before you need it in production. A flag flip that has never been exercised may point at the wrong environment, lack permission, or revert to a prompt version that no longer exists. The drill takes ten minutes and turns the runbook from a document into a verified procedure — the same reason you drill incidents before they happen.