Chaos Engineering and Game Days

"Let's kill a pod on Friday afternoon and see what happens" is not chaos engineering — it is a small outage you scheduled yourself. The discipline is older than the tools and is defined by one requirement: state a falsifiable steady-state hypothesis before you inject anything. Steady state is the normal behavior you can measure continuously — request success rate, latency percentiles, throughput, queue depth, error-budget burn. The experiment tests whether that steady state survives a specific fault. If you cannot say what "healthy" means in numbers, you cannot run an experiment; you can only break things and watch.

A hypothesis you could actually defend in a design review has six parts:

hypothesis:   Checkout success rate stays >= 99.5% and p99 < 800ms
fault:        Terminate one pricing pod (SIGKILL)
scope:        1 of 5 pods, region eu-west-1, synthetic user cohort only
expectation:  Breaker opens <10s, traffic reroutes, recovery <30s
abort_if:     Success rate < 99% for 60s OR page fires for the checkout service
verify:       Grafana "checkout SLO" board + synthetic probe at 1 Hz

Note what is absent: the word "should" doing vague work. Every clause is a number someone can be wrong about — which is the point. The experiment has three possible outcomes and two of them are valuable: the hypothesis holds (you have evidence, not confidence, that the safety property works), or it fails and you find the gap while a customer is not involved. The third outcome is the one practitioners fear most: the fault landed, real damage occurred, and your dashboards showed nothing — a null result that actually means your observability cannot see the failure. Chaos experiments validate detection as much as resilience; "did the right alert fire?" is a first-class success criterion.

Chaos also sits in a specific spot relative to the testing you already do, and interviewers ask for that map:

A staging environment is a reasonable place for early experiments, but it lies in three predictable ways: synthetic traffic is smaller and less varied, your staging topology has drifted from production (one replica instead of six, no cross-AZ failure modes), and nobody's real money is on the line so nobody reads the dashboards. The claims that matter — "the breaker opens under real load," "auto-scaling saves us," "the CDN absorbs an origin failure" — can only be verified where that machinery is actually running. Start there, with a tiny blast radius, which is the next chapter's subject.

Practically, begin by writing hypotheses about protections you already believe exist: "if the primary database fails, reads continue from the replica within 15 seconds," "if one availability zone loses power, capacity drops 40 % but p99 stays under target," "if the payment provider is unreachable, checkout degrades to the queue-and-reconcile path rather than erroring." Each hypothesis is a claim currently living in someone's head. The experiment converts it into a result, and every result goes into an experiment log — hypothesis, scope, window, outcome, follow-ups — so the organization accumulates a record of what has actually been verified instead of what is generally assumed.

Every experiment has a blast radius: the fraction of real users affected, multiplied by how severely, multiplied by how long. Your entire job in production chaos is to make that product small, bounded, and instantly reversible — the experiment must have a smaller radius than the incident it is trying to prevent, or you have invented a more expensive way to cause the outage you were worried about.

Radius control starts with an escalation ladder you climb one rung at a time, gated by the previous rung's results:

1. Local / dev machine, synthetic traffic          → does the fault even trigger?
2. Staging, production-shaped topology             → do metrics and alerts see it?
3. ONE non-critical instance in prod, off-peak     → single user's request path
4. One availability zone, or one tenant cohort     → real failure domain
5. Multi-AZ or region-scale                        → only after 3 and 4 are boring

Never skip rungs because the change looks small. The failure mode of rung 4 is that your "non-critical" instance shares a database, a connection pool, or a message broker with checkout — dependency graphs are where blast radius leaks. Before injecting, confirm isolation explicitly: which services, tenants, and data stores can this target actually reach, and is the traffic you are about to disturb genuinely replacable?

Inside the run, four controls are non-negotiable:

• An automatic abort, wired to the steady-state metrics. If success rate drops below the abort threshold for 60 seconds, the experiment terminates itself — no human decision, no "let's give it one more minute." During an incident your judgment is compromised, so the kill switch must be code, not a person. A manual big-red-button stop exists too, but as the backup to automation, not the primary.
• A hard time box — often 10 minutes — that stops the experiment even if everything looks healthy. Open-ended experiments drift.
• A prepared rollback that has been tested before the day. kubectl rollout undo, a feature flag off, chaos-mesh experiment deletion, an infrastructure restore command. An untested rollback is a hypothesis about recovery you have not run.
• Containment of who is exposed: synthetic canary users, a feature-flagged cohort, one region, read-only faults before write faults. Degrade reads first; injecting faults into the write path comes later, on higher rungs, with finance's explicit agreement.

The fault menu is wider than "kill the process," and choosing the right one is half the experiment design:

Latency injection is usually the highest-value fault and the most underused: real dependencies rarely die cleanly — they get slow, and slow is what kills thread pools. Killing processes tests your restart path; adding 800 ms tests every timeout, breaker, and shedding threshold you have configured.

Tooling is the least interesting part: Chaos Monkey and Netflix's ChAP for scheduled experiments, Gremlin, LitmusChaos or Chaos Mesh for Kubernetes faults, AWS FIS for managed injection, and honestly kubectl cordon/drain, tc netem, and iptables cover a surprising amount. What separates teams is the surrounding process: approval of the window (never during a launch, Black Friday, or alongside a risky deploy — otherwise you cannot attribute symptoms), a shared experiment record, and the discipline that an experiment without a pre-registered hypothesis, an abort condition, and a rollback is just an outage with better branding.

Automated fault injection proves things about your system. A game day proves things about your organization: whether the on-call engineer notices, whether the dashboard answers the question, whether the runbook's third command still exists, whether the status page gets updated before the support inbox fills. Software fails at 02:00 on a Saturday; the people who respond are a component too, and an unattended pod kill does not test them.

A game day has a deliberate arc, and each stage has an artifact:

1. Pre-brief (the day before). Objectives, scenario, injectors, observers, responders, start time, abort criteria, and comms plan. Everyone knows this is an exercise: the fastest way to destroy trust in the program is a "secret" game day where people believe a real incident is unfolding. What must not be pre-announced is the answer — responders get the symptom, not the diagnosis.

2. Injection and observation. The fault goes in at a scheduled minute. Observers timestamp everything: time to detect (does any alert fire?), time to acknowledge (who gets paged, and is it the right person?), time to first correct hypothesis, time to mitigate. The clock on detection is the single most valuable number you will collect, because nothing else in your tooling tells you how long your customers suffered before anyone knew.

3. Response, with the runbook taken literally. Responders execute the documented steps as written — even when a step is obviously wrong, because "step 4 says run ./deploy-failover.sh, which does not exist in any repository" is one of the exercise's findings, and working around it silently in your head defeats the purpose. This stage routinely uncovers dashboards that need three clicks to find, alerts that fire for the wrong service, runbook steps that reference scripts nobody has, an escalation path naming someone who left the company, and rollback credentials nobody on the shift holds.

4. Comms rehearsal. Draft the status-page update and the customer-facing message under time pressure; incident comms written only during real incidents are written badly.

5. Blameless debrief, same day. What did we detect, what did we miss, which runbook steps failed, what was the actual mitigation vs the documented one. "We were lucky" is a finding, not a dismissal.

6. Findings become tickets with owners and dates. This is the step teams skip, and skipping it is what makes the game day theater. Every discovered gap ends up in the backlog, sized, and tracked to done; the next experiment then re-tests that specific claim. Over two or three cycles your resilience documentation stops being aspirational because each entry has been exercised.

What you measure decides whether the program improves: MTTD, MTTR, alert fidelity (right alert, right person, no false-positive storm), runbook step success rate, and time-to-mitigate for each failure mode. Plot them across game days — that trend is your evidence that the program pays for itself.

The two objections share a shape. "We cannot experiment in production" — real traffic shape, data volume, topology, and accumulated config exist only there, so testing elsewhere proves a milder claim; the answer is small blast radius, not abstention. "It's too risky" — unmeasured risk is still risk, and hope without a hypothesis is already an experiment, just an uncontrolled one.

To introduce the practice on a team that has never done it: start in staging with one hypothesis about a protection you already claim to have, small blast radius, one hour, a written log. Then move to a single non-critical instance in production during off-peak. Then schedule quarterly game days with humans in the room. The claims you will be verifying are the ones you built across this whole phase — that your degradation ladder holds (Graceful Degradation and Backpressure), that your breakers trip and recover, that your shed thresholds fire before the OOM killer does — and in an interview, describing an experiment with a hypothesis, a blast radius, an abort criterion, and a follow-up ticket is the most credible way to prove you have actually run one. The same habit — write the claim down, attack it on purpose, track what you found — is exactly what Architecture-Level Threat Modeling formalizes next, where the failures are adversarial instead of accidental.