Inter-service Communication

A checkout screen must display the customer's loyalty tier before it can compute a discount. The user is waiting. This is a synchronous request-response interaction: the caller needs an answer now, and "the other service is down" must become a defined behavior, not a hang. Synchronous REST or gRPC calls suit request-response flows with immediate answers — the work is to pick the right one and to set the deadline before the network picks it for you.

The choice has one honest axis: how strict is the contract, and who is calling?

Both carry the discipline that matters more than the protocol: every outbound call carries a timeout, derived from a deadline, and the deadline shrinks as it propagates. A concrete policy looks like this:

user-facing request budget      800 ms total
checkout's own share            200 ms  → 600 ms left for dependencies
  loyalty-service call          deadline = 600 ms, connect timeout 100 ms
  payments-service call         deadline = 400 ms (it sits behind loyalty)
on timeout                      return a defined degradation (hide the discount),
                                never propagate the hang

The rule of thumb that prevents outages: the downstream timeout must be strictly smaller than the caller's remaining budget. Equal values mean one slow hop consumes everything downstream of it; larger values are how a 10-second worst case appears in a 500-millisecond UI.

Error semantics come with the same rigor. HTTP gives you a vocabulary — 4xx means the caller is wrong (do not retry), 5xx and timeouts mean maybe (retry only with idempotency). That distinction is the seed of the retry policy you will need for every call you add. And because a timeout is not proof of failure (the write may have committed), the request that matters — charging a card, reserving stock — carries an idempotency key so a retry cannot double-charge.

Two supporting decisions round out synchronous calls. Connection management: HTTP/2 multiplexing or a pooled client, because TLS handshakes per call quietly add 50–100 ms. Contract versioning: consumers deploy at different times, so additive-only changes in REST (new optional fields) and Protobuf field-tag discipline in gRPC are not optional hygiene — they are what stops a deploy from becoming a cross-team negotiation.

Interview framing: for "how do services talk?" do not answer "REST is best" or "gRPC is faster." Give the decision rule — caller needs an answer → sync, pick protocol by contract strictness and audience — then immediately add the deadline and idempotency rules, because the question behind the question is always what happens when the other side is slow. That answer opens directly into asynchronous messaging, which solves a different problem: when the caller must not wait at all.

An order is placed and six follow-ups must happen: charge the card, print the label, email the receipt, update analytics, notify the warehouse, bump the loyalty points. Charging must be synchronous — the customer needs to know. The other five must not sit in the user's request path: if the label printer is slow, checkout should not slow down. Asynchronous messaging decouples producers from consumer availability and throughput, and it is worth being precise about which couplings it removes and which it leaves.

What a broker removes: temporal coupling (the producer does not care whether the consumer is running right now — the message waits), throughput coupling (a burst of 10,000 orders drains at the warehouse consumer's pace instead of stalling the producer), and deploy coupling (the consumer can ship its new logic tomorrow; the contract is the event's schema, not the method signature). What it does not remove: the contract still must evolve compatibly, the consumer must still handle duplicates, and business outcomes still need reconciliation — an event published is not an outcome achieved.

The design checklist for a real async flow:

delivery        at-least-once (broker redelivers on crash) → consumer MUST be idempotent
                dedup key = (consumer, message_id) or business key (order_id, attempt)
ordering        per-entity key (order_id) on a partition → per-order order preserved,
                global order not promised — design accordingly
backpressure    consumer lag alert when queue depth grows; scale consumers, not producers
poison messages max receives → dead-letter queue with replay tooling; never loop forever
schema          versioned event schema, additive-only; registry (Protobuf/Avro/JSON Schema)
reconciliation  periodic job: orders without shipped events → investigate. Trust, verify.

The idempotency line deserves emphasis because it is where message systems bite. A broker that guarantees at-least-once will, eventually, deliver the same OrderPlaced twice — after a rebalance, a network blip, a consumer crash between processing and offset commit. A consumer that charges on every delivery will double-charge, and the incident review will discover the delivery semantics for the first time. The durable fix is a processed-messages table keyed by message id, checked inside the same transaction as the business write: exactly-once effects built from at-least-once delivery, which is the only form of exactly-once that survives a real system.

Choice of channel follows the conversation shape. Point-to-point queue: one message, one consumer — work distribution (label printing, job execution). Pub-sub topic: one message, every interested consumer — facts that many parts of the business care about (OrderPlaced read by analytics, billing, and warehouse independently). The tell: if adding a new consumer should require zero changes in the producer, you want a topic; if work should be divided among workers, you want a queue. The patterns are covered in depth later in the roadmap — the communication principle here is that the decision to go async comes from the interaction, not from fashion.

One more honest cost: debugging. A request path has a stack trace; an event chain has a story scattered across three services' logs. Correlation ids on every message (carrying the originating request id) are the price of admission, and a trace that follows messages through the broker is what makes a 2 a.m. "where did this order go?" solvable. Choose async because the caller must not wait — and pay the observability bill so you can still find out what happened.

The checkout path looks fine in the diagram: api-gateway → checkout → loyalty → risk → pricing → inventory. Five arrows, each one reasonable. Now do the arithmetic. If each hop averages 40 ms and the chain is sequential, the caller sees 200 ms of network time before business logic finishes — and under load, tail latency compounds: at 5% chance per hop of a 300 ms stall, the chain has roughly a 23% chance of at least one stall (1 - 0.95^5). Chained synchronous calls create latency multiplication and cascading failure risk — two problems with different fixes.

Latency is fixed by structure. First, fan out in parallel whenever hops do not need each other's results: loyalty, risk, and pricing are independent facts about one order — issue them concurrently and the chain becomes the slowest hop, not the sum (Promise.all / asyncio.gather / gRPC parallel calls): 40 + 40 + 40 becomes ~45 ms. Second, propagate the deadline, not the timeout: the gateway's 800 ms deadline travels as a header (grpc-timeout, x-deadline-ms), each hop computes its own smaller budget from what remains, and a hop that receives an already-expired deadline fails fast instead of burning another 40 ms the caller no longer has. Third, cut the chain: if checkout only needs the result of analytics updates, make them async; if inventory reservation is synchronous but warehouse notification is not, the chain gets shorter where it counts.

Cascading failure needs its own countermeasures, because the second problem is that a chain is only as available as its most fragile member, multiplied by its retry policy. Say inventory is up but slow (2 s): every checkout call waits, threads pile up, checkout's connection pool exhausts, and then its callers start timing out — a failure traveling upward against the call direction. The defense stack, in the order you would deploy it:

1. tight deadlines        (above) — stop the pile-up at the source
2. circuit breaker on each hop — when loyalty's error rate crosses the threshold,
   stop calling it for 30 s; fail fast with the degraded answer (discount hidden)
3. retries with limits    — only on 5xx/timeouts, ≤2 attempts, jittered backoff,
   ONLY because requests carry idempotency keys
4. bulkheads              — separate connection pools per dependency, so one slow
   peer cannot exhaust the pool shared with everything else

Note what the breaker implies: the design already decided how to degrade. "Loyalty down → checkout continues without discounts and flags orders for later repricing" is a product decision made in the design review, not a catch (e) { throw e } discovered in an incident. Services whose failure is genuinely intolerable (payment authorization) get no breaker fallback — they fail the request honestly, and that asymmetry should be written down per dependency in the table from the decomposition workshop.

The systemic view to state in an interview: sync chains trade operational simplicity (one obvious request path, easy to trace) for temporal coupling — every member must be up now, for every request. The correct response is not "make everything async"; it is to (a) keep sync where an answer is required, (b) make the chain as short and as parallel as physics allows, and (c) put budgets, breakers, and degraded answers on every remaining hop. Then measure: track per-hop p99 contribution and you will find the hop that deserves either a cache, a redesign, or a topic — usually one. That measurement habit is what carries into the data story next: with communication chosen, the question becomes who owns the tables these services write to.