Fallacies of Distributed Computing

A payments integration ships with this code: charge(card) and assume it returns. On day 40, a switch reboots mid-request — the TCP connection resets after the charge was committed but before the response arrived. The client sees a timeout, treats it as failure, and retries with a new idempotency key. Two charges, one order, one very unpleasant customer email. The bug was not in the code; it was in an assumption. Assuming the network is reliable ignores partitions, timeouts, and partial failures.

The first fallacy — "the network is reliable" — is the load-bearing one, because accepting it forces you to design for the actual semantics: in a distributed system, failure is partial and information is uncertain. A local function call either returns or throws, and the throw is true. A network call can: never arrive; arrive but the reply is lost; arrive and succeed but the response times out; succeed slowly enough that the caller gave up. The timeout case is the killer — it is not "the call failed," it is "you do not know what happened." Every retry policy must be built on that ignorance.

The resolution is that you never get certainty from the transport; you build it above:

1. idempotency keys    the receiver dedupes, so a retry after an ambiguous timeout
                       is safe (charge(key=k, ...) twice → one charge)
2. status queries      "GET /charges/k" to resolve the unknown — the caller that
                       timed out asks instead of guessing
3. reconciliation      a background job compares intended vs actual outcomes;
                       it is the safety net for every case (1) and (2) missed
4. bounded timeouts    every remote call has one; an open socket is not a plan

Notice the order: prevention (idempotency), resolution (query), and a backstop (reconciliation). Systems that skip the backstop work until the day two of the rare paths collide — and the incident review discovers there was no ledger of intended-vs-actual outcomes to consult.

The fallacy has a human face too: "the network is reliable" usually means "our last 90 days of graphs looked smooth." Smooth graphs measure yesterday's switch firmware. The design question is not whether the network has been kind, but what your system does on the first unkind day — and the answer must be visible in code: timeouts with budgets, retries only where idempotency exists, and a defined degraded path. Note what this interacts with: the four defenses above are the same machinery from Inter-service Communication, because that topic was the how and this one is the why.

Interview framing: "what do you do when a remote call times out?" is the canonical question for this topic. Weak answers list retry counts. Strong answers say the timeout is undecidable, then walk the ladder — dedupe key so retry is safe, query endpoint to resolve, reconciliation as backstop — and finish with the observation that without idempotency you must not retry at all, because a blind retry trades a slow request for a duplicate side effect. That answer shows you have thought about the second-order effect, which is what the next fallacy (latency) trains you to do about time itself.

A team co-locates their API and database — in different availability zones of the same region — and is puzzled that p99 grew 12 ms after the migration. Another team builds a global product that calls a single US-East database from Europe and measures a 140 ms floor on every request. Both fell for the same pair of fallacies: treating latency as zero ignores the real cost of crossing availability zones or regions, and assuming topology never changes ignores that "where the packet goes" is decided fresh every time.

The numbers are worth memorizing — they convert vague placement debates into arithmetic:

Two lessons hide in the table. First, latency multiplies along sequential chains and applies per round trip, not per byte — a chatty API doing ten sequential queries at 1 ms RTT costs more than one batched query at 80 ms. That is why round-trip reduction (batching, co-location, connection reuse) dominates micro-optimization in distributed paths. Second, cross-AZ is a real 1–2 ms tax per hop, and on an internal service mesh doing dozens of hops it becomes a measurable p99 regression — the data above explains both incident reports.

Topology (fallacies 4 and 5 — "topology doesn't change" and "there's one administrator") compounds this. The path your packets take is chosen by BGP routes, load balancers, DNS with TTLs, and a service mesh's retry-and-routing policy — any of which can change while your p99 graph is being admired. A "rare" region evacuation, a failover to the secondary AZ, a NAT gateway replacement: each one silently alters hop counts. And "one administrator" was always fiction — your traffic crosses your admin's domain, your cloud provider's fabric, transit providers, and the destination's network, none of whom coordinate their maintenance windows with yours. Designs that treat a measured RTT as a constant break on the day routing changes; designs with budgets and headroom absorb it.

The practical stance: measure, then budget. Record p99 RTT per dependency (not averages — the tail is the topology changing), give each call a deadline derived from the user-facing budget as covered earlier, and put placement where the arithmetic points: the transaction-critical path stays in one AZ or region, analytics crosses the ocean at night. When an interviewer asks "would you put the database in the same region as the app?", answer with the RTT table and the chain multiplication — not with "yes, because locality" — because the second half of the question is always "and what if a zone fails?", which is the subject of failure models later in this phase.

Four fallacies get less airtime than the first three and bite just as often. They are worth one chapter each in your head and one paragraph in any design review.

"Bandwidth is infinite" (fallacy 3) and "transport cost is zero" (fallacy 6). A link that reads "10 Gbps" is a ceiling under ideal conditions, not a promise to your flow. Real capacity is set by the narrowest segment — a NAT gateway, a load balancer's connection table, an MTU-limited path doing path-MTU discovery on every retry — and it is shared with everything else in the account. Numbers that reset intuition: at 1 Gbps, copying a 10 GB dataset takes 80 seconds of perfect throughput; shipping full snapshots across regions every hour is a budget line, not background noise. Payloads matter too: a JSON document at 3× the size of its binary encoding is 3× the serialization CPU and 3× the bytes competing for the same shared pipe. Bandwidth limits shape payload budgets more than raw link speed suggests — which is precisely why batching and compression decisions deserve a line in the design doc rather than a shrug.

"Security is irrelevant" (fallacy 7). Every hop needs identity: mTLS certificates or signed tokens, key rotation that does not require a midnight all-hands, and least-privilege credentials per service (the database-per-service contract enforced by who may connect, not only by schema). The "zero transport cost" reading of security is literal too: encryption costs CPU and handshake latency — a TLS handshake is 1–2 RTTs, which on a transpacific path is 200–300 ms unless connection resumption or keep-alive is configured. Security that ignores latency gets disabled under pressure; security designed with the RTT table gets kept.

"Reliability is 100%" (fallacy 8). This is the first fallacy's accounting twin: if you believe the network never fails, you conclude your system can be 100% reliable. It cannot — availability composes multiplicatively. Five dependencies each at 99.9% give 0.999^5 ≈ 99.5%, a 36× increase in downtime per year, before your own code is counted. That arithmetic is why degradation paths, retries, and breakers (the machinery from the communication topic) are not optional polish: they are how a 99.9%-per-dependency system earns an acceptable overall number.

The full catalog, compressed for review:

1 network reliable      → timeouts, partial failure, idempotency, reconciliation
2 latency zero          → RTT budgets, batching, co-location, tail measurement
3 bandwidth infinite    → payload budgets, compression, capacity per shared segment
4 topology static       → headroom on RTT SLOs, re-measure after network events
5 one administrator     → cross-provider assumptions, failure domains on the diagram
6 transport cost zero   → TLS/connection overhead in budgets, serialization CPU
7 security irrelevant   → identity per hop, rotation without outages, key management
8 reliability 100%      → multiplicative availability math, degradation by design

Design reviews get sharper when you can name the fallacy a proposal leans on: "this assumes latency is zero" ends an argument faster than "this feels fragile." That identification skill — spot the violated assumption before proposing the architecture — is the transferable one, and it sets up the next topic, where the estimates stop being remembered numbers and start being calculated ones.