Quantifying Latency and Bandwidth

A product manager writes "the page must load in 500 ms" on the ticket. Someone replies "the API is fast, it's like a millisecond" — and the deploy misses the target by 300 ms. The gap is arithmetic nobody performed. A same-datacenter round trip costs hundreds of microseconds while an ocean crossing costs ~150 ms, and the design's job is to turn a user-facing number into a per-hop deadline before the code is written.

Start with the reference numbers. Latency scales over eight orders of magnitude, and the only useful question is always "which order of magnitude is this operation?"

Now the worked budget. The 500 ms page includes the browser: DNS, TLS, HTML, the API call chain, render. Allocate before designing:

user budget            500 ms
browser overhead       100 ms  (DNS 20–40 + TLS 1–2 RTT + HTML parse/first byte)
usable for APIs        400 ms
  API gateway          40 ms   (entry, auth)
  service hop 1        120 ms  (its own downstreams live inside this)
  service hop 2        120 ms
  serialization/render 120 ms
rule: every downstream's deadline < caller's remaining budget — strictly

Two disciplines make the table real. The deadline shrinks hop by hop (covered as policy earlier — here it is simply math: a hop given more than its share can spend time the user no longer has). And the budget is a p99 statement: planning on the mean guarantees missing the target for one request in twenty, which on a busy site is thousands of disappointed users an hour.

The counter-move to a big number is not "buy faster hardware" — it is to reduce round trips, because each RTT is paid serially when the next call depends on the previous answer. A 140 ms transpacific floor times three sequential queries is 420 ms of pure waiting, and no amount of server CPU helps. The three levers, in the order they are cheap: batch (one query returning three rows instead of three queries: 3 RTT → 1), fan out (independent calls in parallel: sum → max), and co-locate (move compute to the data, or cache the data near the compute — a cached read in-region at 300 µs versus an ocean crossing at 150 ms is a 500,000× difference, which is why CDN and read-replica placement decisions dominate everything else in the latency conversation).

Interview framing: when asked to "make this faster," resist the instinct to profile loops first. Ask what the end-to-end budget is, count the sequential round trips against it, and show where the time is structurally going. Candidates who can do this estimate live — "three sequential cross-AZ calls, that's your 6 ms floor right there" — demonstrate the quantitative reflex architects are hired for, and they are ready for the harder question: what happens when there are ten hops, not three.

A checkout request fans out through an API gateway → BFF → checkout → pricing → promotions → risk → loyalty → inventory → payments → notifications. Ten sequential network hops. Each one individually "takes a millisecond." The p99 for the whole path is 600 ms and nobody can find the bug, because there is no bug — there is a waterfall.

Ten sequential network hops turn a small delay into noticeable user-facing latency. Do the arithmetic with the reference table: at 300 µs per same-datacenter hop, ten hops is only 3 ms — fine. But real chains mix distances: two cross-AZ calls (2 ms each), one call to an external fraud API at 60 ms, and the rest in-AZ. Waterfalls in trace form:

gateway        |--|
bff              |--|
checkout            |--------|        (waits on all below)
  pricing              |---|
  promotions           |---|  parallel (good)
  risk                   |--|--|      (cross-AZ: 2 ms)
  loyalty                  |--|       (40 ms external, parallel)
  inventory              |----|
payments                        |-----|  (60 ms external, MUST be last)
                        total ≈ 40 + 3 + 2 + 40 + 60 + ... ≈ 155 ms of transport+externals

Three corrections, in decreasing order of payoff:

1. Flatten the depth, widen the width. Parallel fan-out converts additive latency into the maximum: pricing + promotions + risk together cost the slowest (2 ms), not 4 ms. The rule — any call whose result does not gate another call runs concurrently — usually removes half the waterfall without moving a single deployment.

2. Cut a layer. The BFF and checkout service do not both need to orchestrate; a forwarding layer that adds a hop plus a serialization is a real 300–600 µs and a second failure mode. Every layer you delete is a round trip returned to the budget — the same "forwarding is not a responsibility" lesson from Layered Architecture, now payable in milliseconds.

3. Move what cannot be parallelized. Two sequential external calls (loyalty then payments, because payments needs the quote) are the hardest 100 ms in the path. Options, in order: ask whether the first call's full result is really needed (often a cached loyalty tier suffices — stale by an hour, nobody cares), then whether payments can quote asynchronously before the user hits "Place order" (many checkout flows pre-authorize), and only then accept the cost and spend it explicitly in the deadline table.

What makes this debuggable in production rather than in your head: trace every request and record per-hop p99 contribution. The first time you plot it, one hop is usually 40–60% of the path — the fraud API, the unbatched query, the missing cache — and it becomes the single highest-leverage change in the system. Alert on budget consumed (trace total vs the 500 ms allocation) rather than per-service latency, because a service can be individually fast and still be structurally in the wrong place.

The interview version of this skill: given a system description, ask three questions — how many sequential round trips, what distances are they, which of them have an answer dependency — then name the reshuffle. That is a complete answer to "how would you reduce latency here?" and it needs no profiler.

The dashboard requests "all orders for the last 30 days" and receives a 42 MB JSON document — 118 MB on the wire after the API's pretty-printing, uncompressed. On a 10 Gbps link that sounds like nothing: 95 ms. On the user's 50 Mbps connection it is 19 seconds, and the browser's main thread spends 400 ms parsing it. The link was never the bottleneck the team thought it was. Bandwidth limits shape payload budgets more than raw link speed suggests, because the budget is set by the narrowest shared segment and by bytes that someone must serialize, transmit, and parse.

Do the conversion first — bandwidth is a rate, size is a size:

time = (size_in_bits + overhead) / effective_throughput

42 MB on 10 Gbps datacenter link   ≈  34 ms  (before TCP slow start, shares, parsing)
42 MB on 50 Mbps home link         ≈   6.7 s
1 MB  on 50 Mbps                   ≈ 160 ms
200 KB on 50 Mbps                  ≈  32 ms   ← the realistic budget for one response

Four multipliers hide in that block. TCP slow start means the first megabyte of a fresh connection moves far slower than the link rate — large one-shot responses on new connections underperform the arithmetic, which is why keep-alive and HTTP/2 multiplexing matter as much as compression. The link is shared: the 10 Gbps figure is split with every other flow in the account, and the true ceiling is often a NAT gateway, an LB's connection table, or a path-MTU quirk. The wire multiplies JSON: a pretty-printed JSON document can be 3–5× the size of the equivalent binary encoding and costs 3–10× the parse CPU. And bytes are paid at both ends — the server that serialized 42 MB spent CPU it could have spent serving other requests.

The design moves, each with its arithmetic:

The decision heuristic to state in review: compute a payload's transfer time on the user's connection and its parse cost on the weakest device you support, then compare against that endpoint's slice of the latency budget. If the response does not fit, the endpoint is broken — regardless of how fast the datacenter link reads on the invoice. This same arithmetic is what justifies GraphQL-style field selection, why list endpoints carry default limits in every mature API, and why "just add a field to the response" is a performance review, not a one-line change. You now have both halves of the quantification habit: round trips set the floor, payload budgets set the ceiling — and the next topic adds what happens when the components themselves start failing.