Fault Models and Failure Modes

Two incidents, same night. In the first, risk-engine segfaults — the container restarts, health checks flip to failing, the load balancer pulls it, dashboards go red in seconds. In the second, a firewall rule change silently drops 30% of packets between the payment service and its ledger: no errors, no restarts, just transactions that sometimes take 8 seconds and a retry storm nobody can explain. The crash was loud and the omission was silent — and the silent one cost four hours.

The vocabulary keeps design discussions precise. A fault is the defect or event at the component level (the crash, the dropped packet); an error is the incorrect internal state it produces (a stale balance, a truncated response); a failure is when the system's delivered service deviates from spec (the checkout 500s). Designing means: deciding which faults you tolerate, which errors you detect before they become failures, and which failures users are allowed to see.

The taxonomy you will be asked to recite:

The insight the table encodes: crash faults stop a node while omission faults silently drop messages or responses — and silence is worse than death because it is ambiguous. A crashed node cannot be confused with a healthy one. A dropped response is indistinguishable from a slow one, which is exactly the undecidable timeout from the fallacies topic, now wearing its formal name. This is why robust protocols carry sequence numbers and idempotency: a gap in sequence numbers proves omission where a timeout only suspects it, and a dedup key makes the retry that resolves it safe.

Practical consequences per class, because classification without response is trivia. Crash: run N ≥ 3 so a majority survives one death (the consensus topic's arithmetic), restart aggressively, but fence the old instance — a recovered node that still believes it holds a lock is a crash-recovery fault turned into corruption; leases and epochs (fencing tokens) deny it. Omission: add checksums and heartbeats so loss is detectable, tune retransmission with idempotent consumers, and monitor gaps rather than errors. Byzantine: raise the redundancy bar (next chapter) — or, far more often, decide honestly that you do not defend against it and say so in the design doc.

Interview framing: "what fault model does your system assume?" is the question hiding behind "how does it handle failure?" A complete answer names the classes you tolerate (crash + omission, with timeout-based detection), the classes you explicitly exclude (Byzantine — we trust our own nodes; the external API is authenticated), and one mechanism per included class (health checks, sequence numbers, fencing). Naming what you exclude is the part candidates miss, and it demonstrates that a fault model is a design decision rather than a mood.

The database replica answers every query — within 3 seconds, when it used to answer within 10 milliseconds. Its health check (a trivial SELECT 1, answered in 1 ms) stays green, the failover never triggers, and 20% of user requests crawl. Then the replica recovers, and the problem "fixes itself" before the on-call engineer finishes describing it. This is a timing fault: a response that succeeds late rather than a clear error — the hardest class to catch, because nothing is technically broken.

Timing faults are dangerous precisely because they violate our detection habits. Crash detectors watch liveness; omission detectors watch gaps; but a slow node passes both. Worse, slowness tends to be partial and correlated: one shard, one query shape, one network path. It spreads through the system as thread-pool exhaustion — callers pile up waiting, their own capacity disappears, and the observed failure surfaces in an unrelated service. Gray failures ("the node is up but degraded") are the incident-report name for this shape, and they are a recurring theme in post-mortems precisely because the obvious telemetry is green.

Detection has to be built deliberately, on time rather than on death:

1. latency SLOs per dependency, at p95/p99 — alert on latency BEFORE errors
   (errors arrive late; slowness is the early signal)
2. heartbeats with expected cadence: 'answered SELECT 1 in 1 ms' AND
   'answered the production query shape in < 150 ms' — check the real workload,
   not the synthetic one
3. saturation signals: thread-pool queue depth, connection-pool wait time,
   GC pause rate — the node is 'healthy' while it waits to even start
4. comparative baselines: one replica 10× slower than its siblings — the fastest
   detector of partial degradation is a peer comparison, not an absolute threshold

The response policy matters as much as detection, and it differs from crash handling. You cannot restart your way out of slowness (the restart often does not reproduce it); instead, load shedding and hedging: reject or queue new work before the queue makes every request miss the deadline (protect the many for the one), and consider a hedged request — after the p95 of normal latency, fire the same read at a second replica and take the first answer — which converts tail latency from "additive" to "near-p95" for idempotent reads. For stateful writers, evict via the same fencing discipline as a crash: drain, then remove from the quorum, because a slow voter in a consensus group is a liveness bug, not a performance one.

Design-review framing: for every dependency, write down what "slow" looks like and what triggers the response — a threshold, an action, and an owner. Teams that only define "down" discover during an incident that their outage had no trigger: the system was never down, just uniformly 8× slower, and every threshold in the runbook looked at error rates that never fired. That gap — failure defined as errors, missing slowness entirely — is the timing-fault lesson, and it sets up the class nobody wants to discuss next: components that do not merely fail, but answer incorrectly.

A Byzantine fault is arbitrary behavior: a node that returns a wrong balance with total confidence, tells two peers different versions of the same fact, or follows the protocol except for the one field that matters. The name comes from the generals coordinating an attack by messenger — some generals may be traitors, and messages may be forged — and the formal result that matters to architects: to tolerate f Byzantine faults among n replicas, you need n ≥ 3f + 1 (and quorums above 2f + 1), versus n ≥ 2f + 1 for mere crash tolerance with a simple majority.

That arithmetic is the whole design decision in one line. Three replicas tolerate one crashed node (majority of 2 survives) but zero liars — with three nodes, a traitor plus an honest-looking split can defeat a two-vote majority you cannot verify. Four replicas tolerate one liar because any correct quorum of three contains at least two honest nodes that agree. Byzantine faults involve arbitrary or malicious behavior and need stronger protocols — PBFT-style quorum exchanges with authentication, signature chains on messages, view changes when a leader misbehaves — each adding rounds of communication and cryptographic overhead to a path that, in the crash-only world, needed one append and a heartbeat.

The architect's first job is therefore to scope the adversary honestly:

Note what the table says about the common case: most internal systems are lying to themselves about needing BFT. A payments cluster inside one VPC with mutual TLS and a hardened supply chain should spend its budget on crash recovery, backups, and reconciliation — layers that also catch the accidental "Byzantine-ish" events (a buggy deploy writing wrong values) better than a BFT library would. Where a system does need to defend against independent trust domains, the honest engineering answer is often not "adopt PBFT" but "reduce the trust surface": authenticate every message, sign audit logs, run cross-checkers that compare outputs from independent implementations, and reconcile against an external source of truth. Diverse implementations matter because two copies of the same buggy code agree with each other perfectly — replication defends against death, not against a shared misconception.

Two clarifications interviewers probe. First, arbitrary does not mean random: a Byzantine node behaves strategically — plausible, consistent, wrong where it counts — which is why simple majorities and "the most common answer wins" heuristics fail. Second, BFT is not automatically "more reliable": it trades liveness complexity (view changes, stricter timing assumptions) for safety under adversarial faults; in partial synchrony, protocols can stall rather than commit wrongly. State both, and add your scoping rule — crash model for a single trust domain, authenticated redundancy plus reconciliation across domains, full BFT only when the participants genuinely cannot be trusted — and you have answered the fault-model question at the level it is actually asked. Next, the mechanism that must work despite all these faults: agreeing on order when there is no referee.