Retry Strategies

Fixed Delay Retry

Fixed delay waits the same amount of time between each retry attempt.

Implementation

def retry_fixed_delay(func, max_retries=3, delay=5):
    """Retry with fixed delay"""
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt < max_retries - 1:
                time.sleep(delay)  # Fixed delay
            else:
                raise e

# Usage
result = retry_fixed_delay(
    lambda: call_external_api(),
    max_retries=3,
    delay=5  # Always 5 seconds
)

Retry Timeline

Attempt 1: Fail → wait 5s
Attempt 2: Fail → wait 5s
Attempt 3: Fail → raise exception

Total time: ~10 seconds

Pros and Cons

Pros	Cons
Simple to implement	May overwhelm failing service
Predictable timing	Not adaptive to load
Easy to understand	May cause thundering herd

When to Use

• Simple retry scenarios
• Known, stable failure patterns
• When service recovers quickly
• Testing and development

Exponential Backoff

Exponential backoff increases the delay between retries exponentially.

Formula

delay = base_delay * 2^attempt

Attempt 0: 1 second
Attempt 1: 2 seconds
Attempt 2: 4 seconds
Attempt 3: 8 seconds
Attempt 4: 16 seconds

Implementation

def retry_exponential_backoff(func, max_retries=5, base_delay=1, max_delay=60):
    """Retry with exponential backoff"""
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt < max_retries - 1:
                delay = min(base_delay * (2 ** attempt), max_delay)
                time.sleep(delay)
            else:
                raise e

# Timeline:
# Attempt 1: Fail → wait 1s
# Attempt 2: Fail → wait 2s
# Attempt 3: Fail → wait 4s
# Attempt 4: Fail → wait 8s
# Attempt 5: Fail → raise exception

With Jitter

import random

def retry_exponential_backoff_jitter(func, max_retries=5, base_delay=1):
    """Exponential backoff with jitter"""
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt)
                jitter = random.uniform(0, delay * 0.1)  # 10% jitter
                time.sleep(delay + jitter)
            else:
                raise e

Why Jitter?

Without jitter:
- All clients retry at same time
- Creates thundering herd

With jitter:
- Clients retry at slightly different times
- Smoother load distribution

Comparison

Strategy	Pattern	Use Case
Fixed	1, 1, 1, 1	Simple, stable
Linear	1, 2, 3, 4	Moderate backoff
Exponential	1, 2, 4, 8	Aggressive backoff
Exponential + Jitter	1±0.1, 2±0.2, 4±0.4	Best practice

Jitter in Retries

Types of Jitter

1. Full Jitter:
   delay = random(0, base * 2^attempt)
   Range: [0, max]

2. Equal Jitter:
   delay = base * 2^attempt / 2 + random(0, base * 2^attempt / 2)
   Range: [half, max]

3. Decorrelated Jitter:
   delay = min(max, random(base, prev_delay * 3))
   Based on previous delay

Implementation

import random

def full_jitter(base, attempt, max_delay=60):
    """Full jitter: random between 0 and max"""
    delay = base * (2 ** attempt)
    return min(random.uniform(0, delay), max_delay)

def equal_jitter(base, attempt, max_delay=60):
    """Equal jitter: half fixed, half random"""
    delay = base * (2 ** attempt)
    half = delay / 2
    return min(half + random.uniform(0, half), max_delay)

def decorrelated_jitter(base, attempt, prev_delay=0, max_delay=60):
    """Decorrelated: based on previous delay"""
    if prev_delay == 0:
        return base
    delay = min(max_delay, random.uniform(base, prev_delay * 3))
    return delay

Why Jitter Matters

Scenario: 100 clients retry simultaneously

Without jitter:
- All retry at T+1, T+2, T+4
- Peak load at each retry point
- May cause cascading failures

With jitter:
- Retries spread across time window
- Smoother load distribution
- Better system stability

AWS Recommended Algorithm

def aws_exponential_backoff_jitter(attempt, base=1, max_delay=20):
    """AWS recommended algorithm"""
    delay = min(max_delay, base * (2 ** attempt))
    return random.uniform(0, delay)

Circuit Breaker + Retry Integration

Pattern

Circuit Breaker + Retry:

1. Closed State:
   - Normal operation
   - Retries enabled
   - Count failures

2. Open State:
   - Failure threshold exceeded
   - Retries disabled
   - Fast-fail immediately

3. Half-Open State:
   - After cooldown
   - Allow one request
   - Test if service recovered

Implementation

class CircuitBreakerRetry:
    def __init__(self, failure_threshold=5, recovery_timeout=30):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = 'closed'
        self.last_failure_time = None
    
    def call(self, func, max_retries=3, base_delay=1):
        """Call with circuit breaker and retry"""
        if self.state == 'open':
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = 'half-open'
            else:
                raise CircuitOpenError('Circuit is open')
        
        for attempt in range(max_retries):
            try:
                result = func()
                self._on_success()
                return result
            except Exception as e:
                if attempt < max_retries - 1:
                    delay = base_delay * (2 ** attempt)
                    time.sleep(delay)
                else:
                    self._on_failure()
                    raise
    
    def _on_success(self):
        self.failure_count = 0
        self.state = 'closed'
    
    def _on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = 'open'

Integration Flow

Request → Circuit Breaker Check
           ↓
         [Open] → Fail fast
         [Closed] → Retry Logic
                      ↓
                    Attempt → Success → Return
                    Attempt → Fail → Wait → Retry
                    All Fail → Circuit Open → Fail

Best Practices

1. Use exponential backoff with jitter as default
2. Set max retry limit to prevent infinite loops
3. Integrate circuit breaker for failing services
4. Log retry attempts for debugging
5. Monitor retry rates for anomalies