Message Queues

Why Message Queues

Message queues enable asynchronous communication between services, providing decoupling, buffering, and reliability.

Problems Solved

Without Message Queue:

Service A ──sync call──→ Service B
   │                        │
   ↓ (if B slow, A waits)   ↓
   Blocked                   Processing

With Message Queue:

Service A → [Queue] → Service B
   │          ↑           │
   ↓ (async) │           ↓
   Free      Buffer    Process when ready

Key Benefits

Benefit	Description
Decoupling	Services don't need to know about each other
Buffering	Absorb traffic spikes
Reliability	Messages persist until processed
Scalability	Consumers can scale independently
Async Processing	Non-blocking operations

Synchronous vs Asynchronous

Synchronous:
User Request → API → Database → API → User Response
(Total: 200ms)

Asynchronous:
User Request → API → Queue → User Response (immediate)
                     ↓
               Background Worker → Database
(User gets response in 10ms, processing happens async)

Queue Patterns

Point-to-Point

Producer → [Queue] → Consumer 1
                    → Consumer 2 (only one gets message)

Use case: Task distribution

Publish-Subscribe

Publisher → [Topic] → Subscriber 1
                    → Subscriber 2
                    → Subscriber 3
(All subscribers get message)

Use case: Event broadcasting

Request-Reply

Client → [Request Queue] → Server
Client ← [Reply Queue] ← Server

Use case: Async RPC

Priority Queue

Producer → [Priority Queue] → Consumer
             ↓
         High Priority (processed first)
         Medium Priority
         Low Priority

Use case: Critical task handling

Dead Letter Queue

Producer → [Queue] → Consumer → [DLQ] (failed messages)
                          ↓
                    Retry/Failure handling

Use case: Error handling

Message Queue Use Cases

1. Order Processing

User places order:
1. Order Service → Queue: OrderCreated event
2. Inventory Service consumes: Reserve stock
3. Payment Service consumes: Process payment
4. Shipping Service consumes: Prepare shipment
5. Notification Service consumes: Send confirmation

All services process independently!

2. Email Sending

User triggers email:
1. App → Queue: SendEmail task
2. Worker processes queue:
   - Rate limiting
   - Retry on failure
   - Batching for efficiency

User gets instant response, email sent async

3. Log Aggregation

Application logs:
1. App → Queue: Log events
2. Consumer → Elasticsearch: Indexing
3. Consumer → S3: Archival
4. Consumer → Datadog: Monitoring

One producer, multiple consumers

4. Data Pipeline

ETL Pipeline:
1. Source → Queue: Raw data
2. Transformer → Queue: Cleaned data
3. Aggregator → Queue: Aggregated data
4. Loader → Database: Final data

Each stage processes async

5. Real-time Analytics

User actions:
1. Frontend → Queue: Click events
2. Stream processor: Real-time aggregation
3. Dashboard: Live metrics
4. Data warehouse: Historical analysis