Load Shedding

When to Shed Load

Load shedding drops requests when system is overloaded to protect core functionality.

When to Shed

1. CPU/Memory Near Limit:
   CPU > 90%
   Memory > 85%

2. Queue Depth High:
   Queue > threshold
   Processing lag increasing

3. Latency Spike:
   Response time > SLA
   Timeout rate increasing

4. Error Rate High:
   Error rate > threshold
   Cascading failures

5. Downstream Degraded:
   Dependency slow/failed
   Circuit breaker open

Decision Matrix

Metric	Normal	Warning	Shed
CPU	< 70%	70-90%	> 90%
Memory	< 70%	70-85%	> 85%
Latency P99	< 100ms	100-500ms	> 500ms
Error Rate	< 1%	1-5%	> 5%
Queue Depth	< 1000	1000-5000	> 5000

Implementation

class LoadShedder:
    def __init__(self, thresholds):
        self.thresholds = thresholds
    
    def should_shed(self, metrics):
        for metric, value in metrics.items():
            if metric in self.thresholds:
                if value > self.thresholds[metric]['shed']:
                    return True
        return False
    
    def get_shed_percentage(self, metrics):
        """Calculate what percentage to shed"""
        max_overload = 0
        for metric, value in metrics.items():
            if metric in self.thresholds:
                shed_threshold = self.thresholds[metric]['shed']
                overload = (value - shed_threshold) / shed_threshold
                max_overload = max(max_overload, overload)
        
        return min(max_overload, 1.0)  # Max 100% shed

Shedding Response

def shed_request():
    return Response(
        status=503,
        body={'error': 'Service temporarily overloaded'},
        headers={'Retry-After': '30'}
    )

Priority-Based Shedding

Priority Levels

Priority Levels:

1. Critical (must serve):
   - Health checks
   - Core transactions
   - Paying customers

2. High (prefer to serve):
   - Authenticated users
   - Important operations

3. Medium (nice to serve):
   - Regular requests
   - Non-critical reads

4. Low (shed first):
   - Analytics
   - Background jobs
   - Non-essential features

Implementation

class PriorityLoadShedder:
    def __init__(self):
        self.priorities = {
            'critical': 0,    # Never shed
            'high': 0.2,      # Shed up to 20%
            'medium': 0.5,    # Shed up to 50%
            'low': 1.0        # Shed up to 100%
        }
    
    def should_shed(self, priority, load_level):
        """Check if request should be shed"""
        if priority == 'critical':
            return False
        
        shed_threshold = self.priorities[priority]
        return load_level > shed_threshold
    
    def get_priority(self, request):
        """Determine request priority"""
        if request.headers.get('X-Priority') == 'critical':
            return 'critical'
        if request.user and request.user.is_premium:
            return 'high'
        if request.method in ['GET', 'HEAD']:
            return 'medium'
        return 'low'

Priority Configuration

load_shedding:
  priorities:
    critical:
      threshold: 0.95  # Only shed at 95% overload
    high:
      threshold: 0.8
    medium:
      threshold: 0.6
    low:
      threshold: 0.4

  rules:
    - match: path=/health
      priority: critical
    - match: user.premium=true
      priority: high
    - match: method=GET
      priority: medium

Adaptive Shedding

class AdaptiveShedder:
    def __init__(self):
        self.metrics_history = []
    
    def calculate_shed_rate(self):
        """Adapt shed rate based on metrics"""
        recent = self.metrics_history[-100:]
        avg_latency = sum(r['latency'] for r in recent) / len(recent)
        error_rate = sum(1 for r in recent if r['error']) / len(recent)
        
        if error_rate > 0.1:  # 10% errors
            return 0.5  # Shed 50%
        elif avg_latency > 1000:  # 1 second
            return 0.3  # Shed 30%
        else:
            return 0  # Don't shed

Load Shedding Implementation

Middleware Implementation

class LoadSheddingMiddleware:
    def __init__(self, app, load_shedder):
        self.app = app
        self.shedder = load_shedder
    
    def __call__(self, environ, start_response):
        # Check load
        metrics = self.get_current_metrics()
        
        if self.shedder.should_shed(metrics):
            # Shed request
            response = Response(
                status=503,
                body='Service temporarily overloaded',
                headers={'Retry-After': '30'}
            )
            return response(environ, start_response)
        
        # Process normally
        return self.app(environ, start_response)

Load Shedding at Different Levels

1. Load Balancer Level:
   - Shed at edge
   - Protect entire service
   - Fast, simple

2. API Gateway Level:
   - Route-based shedding
   - Rate limiting integration
   - More granular

3. Application Level:
   - Feature-based shedding
   - Priority-based
   - Most granular

4. Database Level:
   - Query rejection
   - Connection limits
   - Protect data layer

Monitoring

def monitor_load_shedding():
    metrics = {
        'shed_count': get_shed_count(),
        'shed_rate': get_shed_rate(),
        'load_level': get_load_level(),
        'priority_distribution': get_priority_distribution()
    }
    
    if metrics['shed_rate'] > 0.1:  # 10%
        alert('High load shedding rate')
    
    metrics.gauge('load_shedding.rate', metrics['shed_rate'])
    metrics.counter('load_shedding.count', metrics['shed_count'])

Best Practices

1. Shed early (at edge, not deep in stack)
2. Use priority-based shedding
3. Return 503 with Retry-After
4. Monitor shed rates
5. Test under load
6. Have fallback for shed requests