Fault Tolerance

Redundancy

Redundancy duplicates critical components to survive failures.

Types of Redundancy

1. Hardware Redundancy:
   - Multiple servers
   - RAID storage
   - Redundant power supplies

2. Software Redundancy:
   - Multiple instances
   - Replica sets
   - Load balancers

3. Data Redundancy:
   - Replication
   - Backups
   - Geographic distribution

4. Network Redundancy:
   - Multiple paths
   - Failover routes
   - Redundant NICs

Replication Strategies

1. Master-Slave:
   [Master] → [Slave 1]
            → [Slave 2]
   Writes to master, reads from slaves

2. Multi-Master:
   [Master A] ←→ [Master B]
   Both accept writes, sync

3. Peer-to-Peer:
   [Node A] ←→ [Node B] ←→ [Node C]
   All nodes equal

Replication Factor

Replication Factor = Number of copies

RF=1: No redundancy
RF=2: Tolerates 1 failure
RF=3: Tolerates 2 failures
RF=2f+1: Tolerates f failures

Configuration

replication:
  factor: 3
  strategy: rack-aware
  consistency: quorum
  failover: automatic

Failover

Failover automatically switches to a redundant component when primary fails.

Failover Types

1. Active-Passive:
   [Active] → [Passive]
   Primary fails → Secondary activates

2. Active-Active:
   [Active] ←→ [Active]
   Both handle traffic
   One fails → Other takes full load

3. N+1:
   [Active 1] [Active 2] [Active 3] → [Standby]
   Any fails → Standby takes over

4. N+N:
   [Active 1] [Active 2] [Active 3]
   [Active 4] [Active 5] [Active 6]
   Full redundancy

Failover Process

1. Detection:
   - Heartbeat failure
   - Health check failure
   - Error rate spike

2. Decision:
   - Automatic or manual
   - Consensus for split-brain

3. Switchover:
   - Update DNS/registry
   - Redirect traffic
   - Sync state

4. Recovery:
   - Fix failed node
   - Sync data
   - Rejoin cluster

Implementation

class FailoverManager:
    def __init__(self, primary, secondary):
        self.primary = primary
        self.secondary = secondary
        self.active = primary
        self.health_checker = HealthChecker()
    
    def check_health(self):
        if not self.health_checker.is_healthy(self.active):
            self.failover()
    
    def failover(self):
        if self.active == self.primary:
            self.active = self.secondary
        else:
            self.active = self.primary
        
        self.sync_state()
        self.notify_clients()
    
    def sync_state(self):
        # Sync state from old to new primary
        pass

Split-Brain Prevention

Problem: Two nodes both think they're primary

Solutions:
1. Fencing: Kill old primary
2. STONITH: Shoot The Other Node In The Head
3. Quorum: Require majority for leadership
4. Watchdog: Hardware timeout

Graceful Degradation

What is Graceful Degradation?

Full Functionality → Reduced Functionality → Core Only

When components fail:
- Disable non-essential features
- Maintain core functionality
- Inform users of limitations

Degradation Strategies

1. Feature Toggles:
   Disable features when dependencies fail

2. Fallback Responses:
   Return cached/default data

3. Reduced Quality:
   Lower resolution, fewer results

4. Queue for Later:
   Accept request, process when recovered

Implementation

class GracefulDegradation:
    def __init__(self):
        self.features = {
            'recommendations': True,
            'reviews': True,
            'analytics': True
        }
    
    def get_product(self, product_id):
        product = self.db.get_product(product_id)
        
        # Degrade gracefully
        if self.features['recommendations']:
            product['recommendations'] = self.get_recommendations(product_id)
        else:
            product['recommendations'] = []
        
        if self.features['reviews']:
            product['reviews'] = self.get_reviews(product_id)
        else:
            product['reviews'] = []
        
        return product
    
    def disable_feature(self, feature):
        self.features[feature] = False
        self.notify_degradation(feature)

Degradation Levels

Level	Description	User Impact
L0	Full functionality	None
L1	Minor features disabled	Minimal
L2	Major features disabled	Moderate
L3	Core only	Significant
L4	Error page	Complete

Monitoring

def monitor_degradation(service):
    for feature, enabled in service.features.items():
        if not enabled:
            alert(f'{feature} is degraded')
    
    metrics.gauge('degradation.level', get_degradation_level())