Dead Letter Queue

Dead Letter Queue Purpose

A Dead Letter Queue (DLQ) holds messages that failed processing after maximum retries.

Why DLQ?

Without DLQ:

Queue → Consumer → [Fail] → Retry → [Fail] → Retry → ...
                  (blocks queue, no visibility)

With DLQ:

Queue → Consumer → [Fail] → Retry → [Fail] → DLQ
                                    ↓
                              Main queue unblocked
                              Failed message preserved

Benefits

Benefit	Description
Queue Unblocked	Failed messages don't block others
Visibility	See all failed messages in one place
Debugging	Analyze failure patterns
Manual Handling	Human review possible
Metrics	Track failure rates

DLQ Design

class DLQProducer:
    def __init__(self, main_queue, dlq_queue, max_retries=3):
        self.main_queue = main_queue
        self.dlq_queue = dlq_queue
        self.max_retries = max_retries
    
    def process_with_dlq(self, message):
        retry_count = message.get('retry_count', 0)
        
        if retry_count >= self.max_retries:
            # Move to DLQ
            self.dlq_queue.send({
                'original_message': message['body'],
                'retry_count': retry_count,
                'last_error': message.get('error'),
                'timestamp': time.time(),
                'source_queue': self.main_queue.name
            })
            return
        
        try:
            self.process(message['body'])
        except Exception as e:
            # Requeue with incremented retry count
            message['retry_count'] = retry_count + 1
            message['error'] = str(e)
            self.main_queue.send(message)

DLQ Message Format

{
    "original_message": {"order_id": 123, "amount": 99.99},
    "retry_count": 3,
    "last_error": "Payment gateway timeout",
    "timestamp": 1705312400,
    "source_queue": "orders",
    "consumer_id": "order-processor-1",
    "first_failure_at": 1705312000
}

DLQ Processing

Processing Strategies

1. Manual Review:
   - Human inspects DLQ
   - Fixes issues
   - Reprocesses or deletes

2. Automated Reprocessing:
   - Periodic scan DLQ
   - Retry with backoff
   - Move back to main queue

3. Alert + Ignore:
   - Alert on DLQ messages
   - Log for analysis
   - Auto-expire after TTL

Automated Reprocessing

class DLQProcessor:
    def __init__(self, dlq_queue, main_queue):
        self.dlq = dlq_queue
        self.main = main_queue
    
    def process_dlq(self):
        """Periodic DLQ processing"""
        messages = self.dlq.receive_messages(max_messages=100)
        
        for msg in messages:
            try:
                # Attempt reprocessing
                self.process(msg['original_message'])
                
                # Success - remove from DLQ
                self.dlq.delete(msg)
                
            except Exception as e:
                # Still failing
                if msg['retry_count'] < self.max_dlq_retries:
                    # Increment and keep in DLQ
                    msg['retry_count'] += 1
                    msg['last_error'] = str(e)
                    self.dlq.update(msg)
                else:
                    # Permanently failed
                    self.alert_permanent_failure(msg)
                    self.dlq.delete(msg)

DLQ TTL

# Set TTL on DLQ messages
self.dlq.send(message, ttl=604800)  # 7 days

# Auto-expire old DLQ messages
# Prevents unbounded DLQ growth

DLQ vs Poison Queue

Aspect	DLQ	Poison Queue
Purpose	Failed messages	Permanently bad messages
Retry	Yes	No
Action	Reprocess	Alert + ignore
TTL	Shorter	Longer

DLQ Monitoring

Key Metrics

# DLQ monitoring metrics
def monitor_dlq(dlq_queue):
    metrics = {
        'dlq_depth': dlq_queue.get_message_count(),
        'dlq_rate': get_dlq_messages_per_minute(),
        'failure_rate': get_failure_percentage(),
        'top_errors': get_top_error_types()
    }
    
    # Alerts
    if metrics['dlq_depth'] > 1000:
        alert('DLQ depth high')
    
    if metrics['dlq_rate'] > 100:
        alert('High DLQ message rate')
    
    return metrics

Monitoring Dashboard

DLQ Dashboard:

├── Current Depth: 45
├── Messages/min: 12
├── Top Errors:
│   ├── Timeout: 45%
│   ├── Invalid Data: 30%
│   └── Service Unavailable: 25%
├── Age Distribution:
│   ├── < 1 hour: 30
│   ├── 1-24 hours: 10
│   └── > 24 hours: 5
└── Reprocessed Today: 89

Alerting Rules

Metric	Threshold	Severity
DLQ Depth	> 1000	Warning
DLQ Rate	> 100/min	Critical
Old Messages	> 24 hours	Warning
Error Spike	> 2x baseline	Critical

Root Cause Analysis

def analyze_dlq_patterns(dlq_messages):
    """Analyze DLQ for patterns"""
    error_patterns = {}
    
    for msg in dlq_messages:
        error = msg['last_error']
        if error not in error_patterns:
            error_patterns[error] = {
                'count': 0,
                'examples': []
            }
        error_patterns[error]['count'] += 1
        if len(error_patterns[error]['examples']) < 5:
            error_patterns[error]['examples'].append(msg)
    
    return sorted(error_patterns.items(), 
                  key=lambda x: x[1]['count'], reverse=True)

Best Practices

1. Set DLQ TTL to prevent unbounded growth
2. Monitor DLQ depth and rate
3. Analyze patterns for root cause
4. Alert on spikes in DLQ messages
5. Regular review of DLQ contents