Message Queues and Brokers

A product launch hits at 09:00: 40,000 orders land in 90 seconds against a payment-verification service that can process 900 per second. In the synchronous design the API waits for verification, threads pile up, p99 goes from 120 ms to 11 seconds, the load balancer marks the pool unhealthy, and the whole site 503s — even though the capacity problem (900/s vs 666/s average) was mild. The queue's job is to change the shape of that failure: producers hand work to a broker and return immediately; consumers pull at their own sustainable rate, so the burst becomes visible backlog depth instead of cascading latency.

That reframing is the whole value proposition, and it has three concrete effects:

The cost of decoupling is that you now have two failures to manage instead of one: backlog growth and message loss. A queue does not make work disappear — if the sustained rate is below the sustained arrival rate, the backlog grows without bound and you eventually hit retention limits (Kafka: time/bytes retention deletes oldest messages; RabbitMQ/SQS: memory or queue-length limits push back on producers). The operational skill is watching consumer lag (oldest unprocessed message age) as the primary SLA signal: lag under a few seconds means the buffer is working; lag in minutes means you are borrowing time, and lag approaching retention means you are about to lose work.

Which broker for which shape — the comparison interviewers ask for:

RabbitMQ   smart broker, dumb consumers: routes + acks, per-queue semantics,
           per-message (not ordered-across-queue) delivery. Transient work,
           RPC-style task queues, priority, TTL/dead-letter exchanges.
Kafka      dumb broker, smart consumers: immutable ordered log per partition,
           offsets owned by consumers, retention-based replay. Event streams,
           high throughput, consumers that need history (analytics, CDC, search indexing).
SQS        fully managed simple queue: standard = at-least-once, unordered;
           FIFO = dedup + per-message-group ordering. Work queues with no ops burden.
NATS       lightweight pub/sub + core streams, low-latency fan-out.

Backpressure deserves its own rule: the queue must be the thing that says no, or nothing says no. If producers can publish infinitely fast while consumers crawl, you need either bounded queue length with producer-side throttling or a consumer autoscaling policy keyed on lag. Teams that skip this discover it during an incident: the disk fills, the broker starts refusing publishes, and now the ingress path is down because the processing path was slow — the exact coupling the queue was supposed to remove.

Two further design decisions frame everything downstream: pull vs push (pull lets consumers declare their own rate; push requires broker-side prefetch tuning and risks overwhelming a slow consumer) and delivery semantics (at-most-once loses work, at-least-once duplicates it, exactly-once requires idempotent effects or transactional sinks — covered in Event-Driven Architecture Patterns). An interview answer that says 'the queue decouples producers and consumers' and stops is incomplete; say what it converts: synchronous latency into observable backlog, and coupled scaling into independently sized fleets. Next: how one broker's log design delivers ordering and replay.

RabbitMQ deletes a message after delivery; Kafka does not. A Kafka topic is a set of partitions, each an immutable, append-only log of records with monotonically increasing offsets. Nothing is consumed in a meaningful sense — a consumer group's progress is just a stored offset (__consumer_offsets), and reading means asking a broker for records at or after that position. That single design decision produces the properties teams adopt Kafka specifically for:

partition 7:  [0][1][2][3][4][5]...   append-only, retained for 7 days (or ∞)
                   ^ consumer group 'invoice-workers' last committed offset = 3
worker crash → resumes at 4 (or replays 3 if it didn't commit)
reset --to-earliest → reprocess from 0  — REPLAY without touching the broker
new consumer group → reads the whole retained history (backfill for free)

Ordering guarantee, stated precisely: records are ordered within a partition, not within a topic. A producer sending key order-42 gets the same partition every time (hash of key, or explicit partition assignment), so all events for one order are serialized — which is what makes per-entity state machines safe. Across partitions there is no order, because producers may be appending to different partitions concurrently. Therefore: ordering is a design choice expressed through the partition key. If you need global order, you need one partition, which means one writer-bottlenecked stream capped by one broker's throughput — a trade, not a feature.

Retention converts storage into capability. Records live until retention.ms (default 7 days) or retention.bytes, regardless of consumption, which enables the pattern library built on replay: rebuild a broken Elasticsearch index from topic beginning; add a new consumer (fraud model v2) and backfill a year of history; debug yesterday's incident by re-reading exactly what the service saw. The consumer offset commit cadence is the sharp edge: commit-after-processing gives at-least-once (crash between process and commit ⇒ the record is re-delivered); commit-before-processing gives at-most-once (crash after commit ⇒ the record is skipped). Neither is exactly-once by itself — exactly-once needs the transactional producer/broker or an idempotent sink.

Operational details that appear in real incidents:

• Consumer groups and rebalancing. Group members split partitions; adding a consumer triggers a rebalance that revokes and reassigns partitions, during which nothing is consumed. session.timeout.ms/max.poll.interval.ms decide how long a stalled member blocks its partitions before being evicted — set too high, a hung consumer stalls a whole partition's pipeline for minutes.
• Lag is per-partition, and one slow partition stalls that key's ordering guarantee. consumer lag = last produced offset - last committed offset; alert on time lag, not just count.
• Skew: 64 partitions with keys like sequential user IDs can still cluster onto a few brokers if the key distribution is skewed — kafka-consumer-groups --describe shows it immediately.
• The poison pill: a record that crashes the consumer replays forever from the committed offset. Fix with a retry topic or dead-letter queue after N attempts (RabbitMQ natively; Kafka via a DLT consumer), not by skipping the offset commit.

For interviews, the crisp formulation: Kafka gives you a durable, ordered, replayable log per partition with consumer-owned offsets; ordering scope = partition key; delivery semantics = your commit discipline plus idempotent effects. That sentence separates candidates who used Kafka from candidates who read about it. The alternative shape — broker-managed point-to-point delivery — is the subject of Pub/Sub vs Point-to-Point.

A worker pulls message 714 from an SQS-style queue, charges the card, and the container is killed by the OOM reaper before it deletes the message. Five seconds later the visibility timeout expires and 714 is redelivered; the worker charges the customer twice. Nothing is malfunctioning — this is at-least-once delivery behaving exactly as specified. The acknowledgement mode decides what happens after a consumer crash; your code decides whether a duplicate is a bug.

The three modes, and the failure each one accepts:

Exactly-once deserves skepticism in an interview: it is achievable end-to-end only when the effect is idempotent or transactional. Kafka transactions make produce-and-commit atomic within the broker; a consumer writing to MySQL needs an idempotency key (the record's (topic, partition, offset) tuple is ideal) with INSERT ... ON DUPLICATE KEY UPDATE, or an outbox. Design for at-least-once transport plus idempotent processing — that combination is what people colloquially call exactly-once, and it is the answer that survives follow-up questions.

The queue-side mechanics that create redelivery windows, named concretely:

SQS:          visibility timeout (default 30s) — message invisible while in flight,
              returns unless deleted. Extend with heartbeats (ChangeMessageVisibility)
              for long jobs; zero means immediately visible again.
RabbitMQ:     unacked messages are requeued on channel/connection close;
              nack/reject with requeue=true puts the message back (often at the head,
              which is how a poison message livelocks a consumer — use a DLX instead)
Kafka:        no broker-side ack per message; consumer commits offsets.
              Resume point = last committed offset ⇒ duplicates only for
              uncommitted-but-processed records

From that mechanics, the required consumer-side hygiene:

1. Idempotency first. Deterministic operation IDs, dedupe tables with TTL, or natural keys (INSERT IF NOT EXISTS). Every at-least-once design starts here.
2. Visibility/ack timeout ≥ worst-case job duration, with heartbeat extension — or you get redelivery while the original job is still running (two workers on the same job, not merely a retry).
3. Dead-letter queues after N attempts with the failure reason preserved; a DLQ without an alarm and a replay tool is a landfill.
4. Atomic side effect + ack. Commit the DB write and the offset/ack together where possible (Kafka consumer offset stored in the same transaction as the projection), otherwise ack first (loss) or effect first (duplication).
5. Ordering under redelivery. RabbitMQ requeues to the head; Kafka resumes at an uncommitted offset — so downstream effects must tolerate seeing older messages after newer ones unless you process partitions serially.

In system-design interviews this topic shows up as 'how do you process payments without double-charging?'. The full answer: at-least-once delivery, idempotency keys on the effect, DLQ with alerting for the retry storm case, and a reconciliation job that compares ledger entries against provider records for whatever slips through. That same idempotency discipline is the prerequisite for the transactional workflows in The Saga Pattern.