Retries

Retry Policies

Policy Options

1. Fixed Interval:
   retry every N seconds
   Simple but may overwhelm

2. Exponential Backoff:
   retry with increasing delay
   1s, 2s, 4s, 8s...

3. Exponential + Jitter:
   backoff with randomness
   Prevents thundering herd

4. Linear Backoff:
   1s, 2s, 3s, 4s...
   Moderate increase

Implementation

class RetryPolicy:
    def __init__(self, max_retries=3, strategy='exponential'):
        self.max_retries = max_retries
        self.strategy = strategy
    
    def get_delay(self, attempt):
        if self.strategy == 'fixed':
            return 1.0
        elif self.strategy == 'exponential':
            return min(2 ** attempt, 60)
        elif self.strategy == 'exponential_jitter':
            base = min(2 ** attempt, 60)
            return base * random.uniform(0.5, 1.5)
        elif self.strategy == 'linear':
            return attempt + 1

Retryable Errors

def is_retryable(exception):
    """Determine if error is retryable"""
    retryable = [
        ConnectionError,
        TimeoutError,
        TemporaryFailure,
    ]
    
    non_retryable = [
        AuthenticationError,
        ValidationError,
        NotFoundError,
    ]
    
    return any(isinstance(exception, e) for e in retryable)

Retry Configuration

retry:
  max_attempts: 3
  strategy: exponential_jitter
  base_delay: 1s
  max_delay: 60s
  retryable_errors:
    - ConnectionError
    - TimeoutError

Best Practices

1. Only retry retryable errors
2. Use exponential backoff with jitter
3. Set max retry limit
4. Log retry attempts
5. Monitor retry rates

Idempotency in Retries

Why Idempotency?

Problem:
1. Client sends request
2. Server processes
3. Network fails before response
4. Client retries
5. Server processes AGAIN (duplicate!)

Solution: Idempotent operations
- Same request = same result
- Safe to retry

Idempotency Patterns

# Pattern 1: Idempotency Key
def create_order(order, idempotency_key):
    # Check if already processed
    if db.exists('idempotency', idempotency_key):
        return db.get_result(idempotency_key)
    
    # Process and store result
    result = process_order(order)
    db.store('idempotency', idempotency_key, result)
    return result

# Pattern 2: Unique Constraint
def insert_record(record):
    try:
        db.insert('records', record)
    except DuplicateKeyError:
        pass  # Already exists

# Pattern 3: Conditional Update
def update_record(record, expected_version):
    current = db.get('records', record['id'])
    if current['version'] != expected_version:
        return 'Conflict'  # Already updated
    db.update('records', record)

Idempotency by Operation

Operation	Idempotency Strategy
INSERT	Unique constraint, ignore duplicate
UPDATE	Version check, conditional update
DELETE	Delete if exists (no error)
UPSERT	Natural idempotency

Idempotency Keys

import uuid

class IdempotencyManager:
    def __init__(self, db):
        self.db = db
    
    def execute(self, key, operation):
        # Check if already executed
        if self.db.is_executed(key):
            return self.db.get_result(key)
        
        # Execute and store result
        result = operation()
        self.db.store_result(key, result)
        return result

# Generate idempotency key
key = str(uuid.uuid4())
result = idempotency_manager.execute(key, lambda: create_order(order))

Best Practices

1. Always use idempotency keys for retries
2. Store idempotency results with TTL
3. Make operations naturally idempotent when possible
4. Document idempotency guarantees
5. Test retry scenarios thoroughly

Retry Budgets

What is a Retry Budget?

Retry Budget:
- Maximum retries per time window
- Prevents retry storms
- Limits total retry load

Example: Max 10 retries per minute per service

Implementation

class RetryBudget:
    def __init__(self, max_retries=10, window_seconds=60):
        self.max_retries = max_retries
        self.window = window_seconds
        self.retries = []  # timestamps
    
    def can_retry(self):
        now = time.time()
        # Remove old retries
        self.retries = [t for t in self.retries if now - t < self.window]
        
        # Check budget
        return len(self.retries) < self.max_retries
    
    def record_retry(self):
        self.retries.append(time.time())
    
    def get_usage(self):
        now = time.time()
        self.retries = [t for t in self.retries if now - t < self.window]
        return len(self.retries) / self.max_retries

Integration

class RetryWithBudget:
    def __init__(self, retry_policy, retry_budget):
        self.policy = retry_policy
        self.budget = retry_budget
    
    def call(self, func, *args, **kwargs):
        for attempt in range(self.policy.max_retries):
            try:
                return func(*args, **kwargs)
            except Exception as e:
                if not self.policy.is_retryable(e):
                    raise
                
                if not self.budget.can_retry():
                    raise RetryBudgetExhausted()
                
                self.budget.record_retry()
                delay = self.policy.get_delay(attempt)
                time.sleep(delay)

Retry Budget Guidelines

Service Type	Budget
User-facing	10 retries/min
Internal	100 retries/min
Background	1000 retries/min

Monitoring

def monitor_retry_budget(budget, service_name):
    usage = budget.get_usage()
    
    metrics.gauge(f'{service_name}.retry_budget.usage', usage)
    
    if usage > 0.8:
        alert(f'{service_name}: Retry budget near exhaustion')

Best Practices

1. Set retry budgets per service
2. Monitor budget usage
3. Alert on high usage
4. Adjust budgets based on load
5. Consider cascading retry budgets (parent > child)