Queue Design (HLD)

Why Queues Exist

Message queues decouple the producer (sender) from the consumer (receiver). They allow systems to communicate asynchronously, meaning the sender does not need to wait for the receiver to process the message.

Core Use Cases

1. Decoupling Services

Without queues, Service A calls Service B synchronously. If B is slow or down, A is affected.

Without Queue:
Service A ──sync──► Service B (blocks until B responds)

With Queue:
Service A ──► Queue ◄── Service B
  (A continues immediately)   (B processes when ready)

2. Asynchronous Processing

Tasks that don't need to complete before responding to the user.

• User uploads a video → queue for transcoding → respond immediately "Processing started"
• User places order → queue for payment processing, inventory update, email confirmation

3. Buffering / Load Leveling

When producers are faster than consumers, queues absorb the burst.

Producer rate: 10,000 msg/sec
Consumer rate: 1,000 msg/sec

Without queue: Consumers overloaded, requests fail
With queue: Messages buffered, consumers process at their pace

4. Priority and Scheduling

Route different message types to different consumers with different priorities. Delay messages for future processing (e.g., scheduled emails).

5. Retry and Fault Tolerance

If a consumer fails, the message remains in the queue and can be retried. Dead letter queues capture messages that repeatedly fail.

When NOT to Use Queues

• You need synchronous request-response — use direct API calls
• The message must be processed immediately with strong ordering — queues add latency
• Simple notification — webhooks or pub/sub may be simpler
• Low volume — the operational overhead of a queue system may not be justified

Decision Framework

Factor	Use Queue	Don't Use Queue
Processing time	> 100ms	< 100ms
Coupling	Services should be independent	Tight coupling is acceptable
Reliability	Must not lose work	Best-effort is fine
Peak load	Spikes 10x+ normal	Steady traffic
Order	Eventually consistent is OK	Strict ordering required

Apache Kafka

Architecture: Distributed commit log. Messages are written to partitions and retained for a configurable period (days to weeks).

Key characteristics:

• Throughput: Millions of messages/sec
• Retention: Messages persist after consumption (replay capability)
• Ordering: Guaranteed within a partition
• Delivery: At-least-once (exactly-once with idempotent producers + transactional APIs)
• Consumer model: Pull-based (consumers poll the broker)
• Partitioning: Topics split into partitions for parallelism

Best for: Event streaming, log aggregation, real-time data pipelines, systems needing message replay.

RabbitMQ

Architecture: Traditional message broker with exchanges, queues, and bindings.

Key characteristics:

• Throughput: Tens of thousands of messages/sec
• Retention: Messages deleted after consumption (unless using lazy queues)
• Ordering: FIFO within a queue (if single consumer)
• Delivery: At-least-once or exactly-once (with publisher confirms)
• Push-based: Broker pushes to consumers
• Routing: Flexible routing via exchanges (direct, topic, fanout, headers)

Best for: Traditional task queues, complex routing, RPC patterns, systems needing message acknowledgment.

Amazon SQS

Architecture: Fully managed cloud queue service.

Key characteristics:

• Throughput: Virtually unlimited (auto-scaling)
• Retention: Up to 14 days
• Types: Standard (best-effort ordering, at-least-once) and FIFO (exactly-once, strict ordering)
• Delivery: At-least-once (Standard) or exactly-once (FIFO)
• Max message size: 256 KB (use SQS Extended Client for larger)
• Visibility timeout: Messages hidden from other consumers while being processed

Best for: AWS-native applications, fully managed operation, decoupling microservices.

Comparison Table

Feature	Kafka	RabbitMQ	SQS
Throughput	Millions/sec	Tens of thousands/sec	Unlimited (managed)
Message retention	Configurable (days)	Until consumed	Up to 14 days
Ordering	Per-partition	Per-queue	Standard: best-effort, FIFO: strict
Delivery guarantee	At-least-once	At-least-once	At-least-once or exactly-once
Consumer model	Pull	Push	Long polling
Routing	Consumer-side	Exchange-based	Queue-based
Operations	Self-managed (or Confluent)	Self-managed (or CloudAMQP)	Fully managed
Replay	Yes (messages retained)	No (deleted after ack)	No
Cost model	Per broker node	Per node	Per request + data transfer

Point-to-Point

One message is consumed by exactly one consumer.

Producer ──► [Queue] ──► Consumer A
                    ──► Consumer B (competing consumers)

Both consumers pull from the same queue. Each message goes to only one consumer. This is the simplest model for load balancing work across multiple workers.

Use cases: Task distribution, background job processing.

Pub-Sub (Publish-Subscribe)

One message is delivered to ALL subscribers.

Publisher ──► [Exchange/Topic]
                  ├──► Subscriber A
                  ├──► Subscriber B
                  └──► Subscriber C

Every subscriber receives every message. This is fan-out by nature.

Kafka implementation: Multiple consumer groups each get their own copy of the offset, so each group independently consumes all messages.

RabbitMQ implementation: Fanout exchange or topic exchange with wildcard bindings.

Use cases: Event broadcasting, notifying multiple services of a state change.

Fan-Out

A single event triggers processing in multiple downstream systems.

Order Placed Event
  │
  ├──► Payment Service (charge card)
  ├──► Inventory Service (reserve stock)
  ├──► Notification Service (send confirmation email)
  └──► Analytics Service (record event)

Implemented via:

• Kafka: Multiple consumer groups reading from the same topic
• RabbitMQ: Fanout exchange with multiple bound queues
• SQS: SNS topic fanning out to multiple SQS queues

Consumer Groups

A consumer group is a set of consumers that cooperate to consume messages from a topic.

Rules:

• Each partition is consumed by exactly ONE consumer within a group
• Consumers in different groups each get ALL messages
• Adding consumers to a group (up to the number of partitions) increases parallelism

Topic: orders (3 partitions)

Consumer Group: payment-service
  Consumer 1 ← Partition 0
  Consumer 2 ← Partition 1
  Consumer 3 ← Partition 2

Consumer Group: analytics-service
  Consumer 1 ← Partition 0
  Consumer 2 ← Partition 1
  Consumer 3 ← Partition 2

Message Ordering

• Per-partition ordering (Kafka): Messages in the same partition are ordered. Use a partition key (e.g., user_id) to ensure related messages go to the same partition.
• Per-queue ordering (RabbitMQ/SQS FIFO): Messages in a FIFO queue are strictly ordered.
• Global ordering: Only possible with a single partition/queue (throughput bottleneck).

Delivery Guarantees

Guarantee	Description	How
At-most-once	Message may be lost, never delivered twice	Fire-and-forget, no ack
At-least-once	Message may be delivered multiple times, never lost	Ack after processing, re-deliver on failure
Exactly-once	Message delivered exactly once	Idempotent consumers + transactional offset commits

Exactly-once is hard. In practice, implement at-least-once delivery with idempotent consumers.

Idempotency

A consumer must handle duplicate messages gracefully. Techniques:

• Idempotency keys: Store processed message IDs; skip if already seen
• Database unique constraints: Use UNIQUE constraints so duplicate inserts fail
• Conditional updates: UPDATE ... WHERE version = expected_version
• Natural idempotency: SET status = 'shipped' is idempotent regardless of how many times it runs

Dead Letter Queues (DLQ)

Messages that repeatedly fail processing are moved to a DLQ instead of being retried forever.

Producer ──► [Main Queue] ──► Consumer (fails 3x)
                              │
                              ▼
                        [DLQ] ──► Alert / Manual Review

Configuration:

• Set max retry count (e.g., 3-5 attempts)
• After max retries, move to DLQ
• Monitor DLQ size as an operational metric
• DLQ messages can be replayed after fixing the bug

Retry Policies

Policy	Behavior
Immediate retry	Retry instantly (may cause repeated failure)
Fixed delay	Retry after N seconds
Exponential backoff	Retry after 1s, 2s, 4s, 8s... (with jitter)
Exponential backoff + jitter	Add randomness to prevent thundering herd

Best practice: Exponential backoff with jitter is the standard recommendation.

delay = min(base * 2^attempt + random(0, jitter), max_delay)

Backpressure and Flow Control

When consumers can't keep up with producers:

• Queue size limits: Reject or block producers when queue is full
• Rate limiting: Throttle producers at the API gateway level
• Consumer scaling: Auto-scale consumers based on queue depth
• Buffer overflow to disk: Kafka persists to disk, handling backpressure naturally
• Circuit breaker: Stop accepting messages when downstream is overwhelmed

Producer ──► [Queue: 80% full] ──► Signal: slow down
                │
                ▼
Producer reduces rate or buffers locally

Monitoring Key Metrics

• Queue depth: Number of messages waiting (high = consumers too slow)
• Consumer lag: How far behind consumers are from the latest message
• Processing rate: Messages processed per second
• DLQ depth: Number of failed messages (high = bug or capacity issue)
• End-to-end latency: Time from message enqueue to processing completion

Problem Statement

Design the queue architecture for an e-commerce order processing system. When a user places an order, multiple downstream actions must occur: payment, inventory reservation, shipping calculation, email confirmation, and analytics tracking.

Requirements

• 10,000 orders/hour at peak (Black Friday: 100,000 orders/hour)
• Payment must complete before shipping notification
• Email and analytics can happen asynchronously
• System must not lose any orders
• Failed orders must be retried and eventually land in a DLQ

Architecture

Order Service
  │
  ▼
[Order Created Event] ──► Kafka Topic: orders
  │
  ├──► Consumer Group: payment-service
  │       ├── Process payment
  │       └── On success, emit: Payment Completed Event
  │
  ├──► Consumer Group: analytics-service
  │       └── Record order event (fire-and-forget acceptable)
  │
Payment Completed Event ──► Kafka Topic: payments
  │
  ├──► Consumer Group: inventory-service
  │       └── Reserve inventory
  │
  ├──► Consumer Group: shipping-service
  │       └── Calculate shipping, create shipment
  │
  └──► Consumer Group: notification-service
          └── Send confirmation email

Partitioning Strategy

Partition key: order_id

// All events for the same order go to the same partition
// Ensures ordering of events within an order
partition = hash(order_id) % num_partitions

This ensures that for any given order, the sequence of events (created → payment → shipped) is processed in order.

Message Schema

{
  "event_id": "uuid-1234",
  "event_type": "ORDER_CREATED",
  "order_id": "ORD-5678",
  "user_id": "USER-9012",
  "timestamp": "2024-01-15T10:30:00Z",
  "payload": {
    "items": [{"sku": "ABC", "qty": 2}],
    "total": 49.99,
    "currency": "USD"
  }
}

Handling Failures

Payment fails:

1. Payment service retries 3 times with exponential backoff
2. After 3 failures, emit ORDER_PAYMENT_FAILED event
3. Notification service sends failure email to customer
4. Order status updated to PAYMENT_FAILED

Inventory insufficient:

1. Inventory service attempts reservation
2. If insufficient, emit ORDER_INSUFFICIENT_INVENTORY event
3. Payment service issues refund
4. Notification service notifies customer

DLQ Configuration

Main Topic: orders
  └── DLQ Topic: orders-dlq (retention: 30 days)

DLQ Processing:
1. Alert if DLQ depth > 0
2. Ops team inspects failed messages
3. Fix root cause
4. Replay DLQ messages back to main topic

Scaling

• Partitions: Start with 12 partitions (supports 12 consumers in parallel)
• Consumers: Auto-scale based on consumer lag
  • Normal: 3 payment consumers
  • Peak: Scale to 12 payment consumers
• Kafka brokers: 3-node cluster with replication factor 3

Monitoring Dashboard

Metric	Normal	Alert Threshold
Orders/sec	2-3	> 30 (unexpected spike)
Payment consumer lag	< 100	> 1,000
DLQ depth	0	> 0
End-to-end latency	< 5s	> 30s
Payment failure rate	< 1%	> 5%