Timeouts and Retries

Your checkout service calls the pricing service. Pricing's database starts stalling: every query takes 8 seconds instead of 40 milliseconds. Your checkout code uses a default HTTP client with no timeout, so every request waits patiently. Within a minute, 200 worker threads are parked inside pricing calls, the thread pool is full, and /checkout returns nothing to anyone — not even errors. A pricing slowdown has become a total checkout outage, and nothing in the logs explains it, because no call ever failed. They just never finished.

The fix is a number on every wait. A timeout is a statement about how long you are willing to wait, and every hop in the call chain needs one. Threads and sockets are finite resources held for the entire duration of a call; an unbounded wait is an unbounded leak of both. Worse, the OS-level give-up is slow — Linux TCP retransmission can hold a dead socket for over 15 minutes before erroring — so "the network will eventually fail it" is not a timeout, it is a slow-motion pileup.

Think of it as an inventory of every wait your request performs:

Two stacking rules follow from that inventory, and both are interview favorites.

First, the server-side timeout should be slightly shorter than the client's per-attempt timeout. If the dependency can decide it is stuck, it should answer 503 with a reason while the client is still listening. When the client gives up first, the server keeps executing the work to completion anyway and the caller learns nothing — you pay full price for a request that returns no diagnosis.

Second, the caller's total deadline must cover everything it might actually do: attempts × per-attempt-timeout + sum(backoff sleeps). Three attempts at 2 seconds with exponential backoff is a 7–10 second operation wearing a 2-second label. If your own caller — a browser at 30 seconds, an API gateway at 10 seconds — has a shorter deadline than your retry plan, your late successes are computed and then discarded. You pay for the work twice and observe it zero times.

A working per-attempt client in Go shows how many separate ceilings one "timeout" hides:

transport := &http.Transport{
    DialContext:           (&net.Dialer{Timeout: 300 * time.Millisecond}).DialContext,
    TLSHandshakeTimeout:   1 * time.Second,
    ResponseHeaderTimeout: 1500 * time.Millisecond,
    IdleConnTimeout:       90 * time.Second, // stale keep-alive sockets after an LB restart
    MaxIdleConnsPerHost:   32,
}
client := &http.Client{Transport: transport, Timeout: 2 * time.Second} // total, incl. body

The forgotten one is usually IdleConnTimeout: a load balancer restarts, a keep-alive socket is dead on the peer, and the first request to reuse it hangs until some outer layer gives up on it. A cache miss that has to fetch from a slow backend is the same shape of problem you met in Cache Invalidation Trade-offs — the fetch needs its own deadline too.

Interviewers ask this as "what happens when a dependency is slow rather than down?" Answer with thread-pool exhaustion and cascading failure, never with "we retry" — retries are chapter two's problem, and without a timeout they have nothing to retry on.

A dependency blips for 30 seconds. You have 5,000 instances with 50 in-flight calls each, all configured the same way: on failure, wait a fixed 500 ms and retry once. At t+500 ms, roughly 250,000 requests arrive at the struggling service in the same instant. Its recovery queue, which was draining nicely, is now deeper than the blip; those retries fail too; another synchronized wave follows. The 30-second fault became a multi-minute outage, entirely because of your retry policy.

Fixed delay is the classic mistake: retries must use exponential backoff with jitter, or the fleet retries in lockstep. Backoff grows the wait (base × 2^attempt, capped) so later attempts give the dependency more room. Jitter randomizes when each caller lands inside that window, which is what actually breaks the synchronization — every instance in your fleet shares the same fault, so without jitter they share the same retry instant too.

With base=100 ms and cap=20 s, full jitter gives attempt 3 a draw from [0, 800 ms] and attempt 5 a draw from [0, 3.2 s] — no two callers agree, yet the average load falls exactly as the backoff curve intends. The implementation is four lines:

import random, time

def backoff(attempt, base=0.1, cap=20.0):
    # full jitter: sleep somewhere in [0, min(cap, base * 2**attempt)]
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Backoff alone still does not bound your load, and this is the part most candidates miss. If 1 % of traffic fails and every failed request makes three more attempts, the dependency sees up to 3 % extra traffic at the worst possible moment; if the failure rate is 50 %, it sees a multiple of its normal load while it is already sick. The bound you want is a retry budget: cap retries as a fraction of successful traffic — for example, allow retries totaling at most 10 % of the original request rate in any window, enforced by a token bucket shared per dependency. When the bucket is empty, fail fast and surface the error. Envoy implements exactly this with retry_budget { budget_ratio: 0.2; min_retry_concurrency: 3 } on a cluster; Resilience4j composes Retry with a RateLimiter for the same effect.

Two rules complete the policy. Honor Retry-After: if the server says 30 seconds and your deadline is 5, do not sleep — fail now, because the sleep is guaranteed to be wasted. And check the clock before sleeping: never start an attempt that cannot finish inside the caller's remaining deadline.

Measure what you ship: emit retry attempts as a metric broken out by dependency and outcome, and alert when the retry rate crosses a threshold rather than waiting for the success rate to fall. A rising retry ratio with a still-healthy success rate is the earliest warning that a dependency is degrading and your budget is about to be spent. Track retry success rate too: attempts that keep succeeding on the first retry but never on the third are telling you the backoff is too aggressive for that failure mode.

"How do you prevent a retry storm?" is a standard interview question, and a complete answer names both halves: exponential backoff with jitter spreads the attempts, and a retry budget bounds them. Neither alone is sufficient.

You send POST /v1/charges for $49.99. The socket dies while you are waiting for the response. Did the charge happen? You cannot tell — and if you retry and the server had in fact committed, the customer pays twice. Retries are only free when the operation is idempotent: repeating it produces the same end state. HTTP builds this into the verbs — GET, HEAD, PUT, DELETE, OPTIONS are idempotent by definition, POST is not — but transport semantics are not enough on their own. What you actually need is a map of where an attempt failed and what that implies.

The last three rows are where incident reports get written. A 500 is not automatically "safe to retry": handlers often commit the transaction and then blow up rendering the response.

For non-idempotent operations, the standard fix is an idempotency key, and it is worth stating precisely because interviewers probe it:

POST /v1/charges HTTP/1.1
Idempotency-Key: 2f8f6f1e-8a37-4b64-9d0e-0d1c8f5b1a44
Content-Type: application/json

{"amount": 4999, "currency": "USD", "order_id": "A-10241"}

The client generates one UUID per logical operation, not per attempt, and sends it on every attempt. The server records key → stored response atomically — a unique constraint so concurrent duplicates collide instead of double-charging — and replays the stored status and body for every subsequent arrival inside the TTL, which must exceed your longest retry window (24 hours is the usual contract). The record must live in the same durability class as the transaction itself: a Redis "cache" of keys is not good enough when real money is involved. The first attempt returns 201 Created; the replay returns the identical stored 201, not a second charge.

When the downstream API supports neither idempotent verbs nor keys, stop retrying automatically. Return an explicit ambiguous state — "payment submitted, status unknown" — and reconcile asynchronously by querying status with a client-side identifier. Reconciliation beats blind retrying: it resolves uncertainty instead of doubling side effects. Server-side, you can often make non-idempotent endpoints safe anyway: set-style operations (PUT /carts/7/items/9 with {"qty": 2} rather than "add one"), conditional writes on a version column, and upserts all converge under repetition.

Retries also inherit the deadline rules from the first chapter: an attempt that would start at 1.9 s under a 2 s deadline should never start. And they inherit a load ceiling from chapter two — which is why the natural next step is Circuit Breaker Pattern: when the dependency is genuinely down, every retry is guaranteed wasted work, and the breaker decides how many of them you are allowed to burn before failing fast.