Heartbeats

Heartbeat Mechanism

Heartbeats are periodic signals indicating a node is alive and functioning.

How Heartbeats Work

Leader → Heartbeat → Follower
Leader ← Ack ← Follower

If no heartbeat within timeout:
- Node considered dead
- Trigger failover/re-election

Implementation

import threading
import time

class HeartbeatSender:
    def __init__(self, interval=5):
        self.interval = interval
        self.running = False
    
    def start(self, callback):
        self.running = True
        def loop():
            while self.running:
                callback()
                time.sleep(self.interval)
        
        thread = threading.Thread(target=loop, daemon=True)
        thread.start()
    
    def stop(self):
        self.running = False

class HeartbeatReceiver:
    def __init__(self, timeout=15):
        self.timeout = timeout
        self.last_heartbeat = time.time()
        self.lock = threading.Lock()
    
    def receive(self):
        with self.lock:
            self.last_heartbeat = time.time()
    
    def is_alive(self):
        with self.lock:
            return (time.time() - self.last_heartbeat) < self.timeout

Heartbeat Patterns

Pattern	Description	Use Case
Ping-Pong	Request-response	Simple health check
One-way	Unidirectional	Low overhead
Gossip	Peer-to-peer	Large clusters
Centralized	To coordinator	Simple topology

Failure Detection

Detection Methods

1. Timeout-based:
   - Miss N heartbeats = failure
   - Simple, common

2. Quorum-based:
   - Multiple nodes agree
   - More accurate

3. Phi Accrual:
   - Probabilistic detection
   - Adaptive threshold

Implementation

class FailureDetector:
    def __init__(self, phi_threshold=8):
        self.phi_threshold = phi_threshold
        self.heartbeats = []
    
    def heartbeat(self):
        self.heartbeats.append(time.time())
        # Keep last 100 heartbeats
        if len(self.heartbeats) > 100:
            self.heartbeats.pop(0)
    
    def phi(self):
        """Calculate phi (probability of failure)"""
        if len(self.heartbeats) < 2:
            return 0
        
        # Calculate inter-arrival times
        intervals = [
            self.heartbeats[i+1] - self.heartbeats[i]
            for i in range(len(self.heartbeats) - 1)
        ]
        
        # Mean interval
        mean = sum(intervals) / len(intervals)
        
        # Time since last heartbeat
        elapsed = time.time() - self.heartbeats[-1]
        
        # Phi = -log10(P_later)
        # P_later = e^(-elapsed/mean)
        p_later = math.exp(-elapsed / mean)
        phi = -math.log10(p_later)
        
        return phi
    
    def is_failed(self):
        return self.phi() > self.phi_threshold

Detection Accuracy

Method	False Positives	False Negatives	Complexity
Timeout	High	Low	Low
Quorum	Low	Low	Medium
Phi Accrual	Low	Low	High

Timeout Configuration

Configuration Parameters

1. Heartbeat Interval:
   - How often to send heartbeat
   - Typical: 1-5 seconds

2. Failure Timeout:
   - How long to wait before declaring dead
   - Typical: 10-30 seconds

3. Retry Interval:
   - How often to retry failed nodes
   - Typical: 5-10 seconds

Configuration Example

heartbeat:
  interval: 3s        # Send every 3 seconds
  timeout: 15s        # Declare dead after 15s
  retry_interval: 5s  # Retry every 5 seconds
  max_missed: 5       # Miss 5 = failure

Timing Diagram

Sender:     |--H--|--H--|--H--|--H--|--H--|--X--|--X--
Receiver:   |--R--|--R--|--R--|--R--|     |     |
            ↑                 ↑           ↑
          Start           Last OK     Timeout
                                        (dead)

Guidelines

Scenario	Interval	Timeout
LAN	1s	5-10s
WAN	3-5s	15-30s
Cloud	1-3s	10-20s
High-latency	5-10s	30-60s

Best Practices

1. Interval < Timeout/3 (detect before timeout)
2. Account for network latency
3. Use phi accrual for adaptive detection
4. Monitor false positive rate
5. Adjust based on conditions