Circuit Breaker

Circuit Breaker Three States

State Diagram

Closed → Open → Half-Open → Closed
  ↑                    ↓
  └────────────────────┘

Closed:
- Normal operation
- Requests pass through
- Count failures

Open:
- Failure threshold exceeded
- Requests fail immediately
- No calls to service

Half-Open:
- Cooldown period elapsed
- Allow one test request
- Check if service recovered

State Transitions

1. Closed → Open:
   Trigger: Failure count >= threshold
   Action: Start cooldown timer

2. Open → Half-Open:
   Trigger: Cooldown period elapsed
   Action: Allow one test request

3. Half-Open → Closed:
   Trigger: Test request succeeds
   Action: Reset failure count

4. Half-Open → Open:
   Trigger: Test request fails
   Action: Reset cooldown timer

Configuration

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.last_failure_time = None
        self.state = 'closed'

State Impact

State	Request Behavior	Failure Detection
Closed	Pass through	Count failures
Open	Fail fast	No detection
Half-Open	Allow one	Test recovery

Circuit Breaker Implementation

Complete Implementation

import time
import threading

class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30, half_open_max=1):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max = half_open_max
        
        self.failure_count = 0
        self.success_count = 0
        self.last_failure_time = None
        self.state = 'closed'
        self.lock = threading.Lock()
    
    @property
    def is_open(self):
        with self.lock:
            if self.state == 'open':
                if time.time() - self.last_failure_time > self.recovery_timeout:
                    self.state = 'half-open'
                    self.success_count = 0
                    return False
                return True
            return False
    
    def record_success(self):
        with self.lock:
            self.failure_count = 0
            if self.state == 'half-open':
                self.success_count += 1
                if self.success_count >= self.half_open_max:
                    self.state = 'closed'
    
    def record_failure(self):
        with self.lock:
            self.failure_count += 1
            self.last_failure_time = time.time()
            
            if self.state == 'half-open':
                self.state = 'open'
            elif self.failure_count >= self.failure_threshold:
                self.state = 'open'
    
    def call(self, func, fallback=None):
        if self.is_open:
            if fallback:
                return fallback()
            raise CircuitOpenError('Circuit is open')
        
        try:
            result = func()
            self.record_success()
            return result
        except Exception as e:
            self.record_failure()
            if fallback:
                return fallback()
            raise

# Usage
circuit = CircuitBreaker(failure_threshold=5, recovery_timeout=30)

try:
    result = circuit.call(lambda: call_external_service())
except CircuitOpenError:
    result = fallback_response()

With Fallback

class ResilientService:
    def __init__(self):
        self.circuit = CircuitBreaker()
    
    def get_user(self, user_id):
        try:
            return self.circuit.call(
                lambda: self.primary_db.get_user(user_id)
            )
        except CircuitOpenError:
            # Fallback to cache or secondary
            return self.cache.get_user(user_id)

Monitoring

def monitor_circuit_breaker(circuit, service_name):
    metrics = {
        'state': circuit.state,
        'failure_count': circuit.failure_count,
        'is_open': circuit.is_open
    }
    
    metrics.gauge(f'{service_name}.circuit.state', 
                  ['closed', 'open', 'half-open'].index(circuit.state))
    metrics.counter(f'{service_name}.circuit.failures', circuit.failure_count)

Best Practices

1. Set appropriate thresholds based on service
2. Use fallbacks when circuit is open
3. Monitor circuit state
4. Log state transitions
5. Test failure scenarios

Circuit Breaker + Retry Integration

Combined Pattern

Request → Circuit Breaker Check
           ↓
         [Open] → Fail fast (no retry)
         [Closed] → Retry Logic
                      ↓
                    Attempt → Success → Return
                    Attempt → Fail → Backoff → Retry
                    All Fail → Record Failure → Circuit Open

Implementation

class ResilientCaller:
    def __init__(self, circuit_breaker, retry_policy):
        self.circuit = circuit_breaker
        self.retry = retry_policy
    
    def call(self, func, *args, **kwargs):
        # Check circuit first
        if self.circuit.is_open:
            raise CircuitOpenError('Circuit is open')
        
        # Retry with backoff
        last_exception = None
        for attempt in range(self.retry.max_retries):
            try:
                result = func(*args, **kwargs)
                self.circuit.record_success()
                return result
            except Exception as e:
                last_exception = e
                self.circuit.record_failure()
                
                if attempt < self.retry.max_retries - 1:
                    delay = self.retry.get_delay(attempt)
                    time.sleep(delay)
        
        raise last_exception

Flow with Integration

1. Request arrives
2. Circuit open? → Yes → Fail fast
3. Circuit closed? → Continue
4. Attempt call
5. Success → Record success, return
6. Failure → Record failure
7. More retries? → Wait (backoff) → Retry
8. No more retries → Raise exception
9. Circuit opens if threshold exceeded

Configuration Example

# Configuration
circuit_config = {
    'failure_threshold': 5,
    'recovery_timeout': 30
}

retry_config = {
    'max_retries': 3,
    'base_delay': 1,
    'max_delay': 30,
    'jitter': True
}

# Create components
circuit = CircuitBreaker(**circuit_config)
retry = RetryPolicy(**retry_config)
caller = ResilientCaller(circuit, retry)

# Use
result = caller.call(service_api, param1, param2)

Monitoring

def monitor_resilience(caller, service_name):
    metrics = {
        'circuit_state': caller.circuit.state,
        'failure_count': caller.circuit.failure_count,
        'retry_rate': get_retry_rate(service_name)
    }
    
    # Alert on open circuit
    if caller.circuit.is_open:
        alert(f'{service_name}: Circuit is open')

Best Practices

1. Use circuit breaker first (fast fail)
2. Retry only when circuit closed
3. Use fallbacks when circuit open
4. Monitor both metrics
5. Test failure scenarios