Reliability Design

SLA, SLO, and SLI

SLI (Service Level Indicator) — A quantitative measure of a specific aspect of service behavior.

• Latency: p99 request latency < 200ms
• Throughput: successful requests per second
• Error rate: percentage of failed requests
• Availability: fraction of time service is operational

SLO (Service Level Objective) — An internal target value for an SLI over a specified time window.

• "99.9% of requests complete in under 200ms over a 30-day rolling window"
• "Error rate must stay below 0.1% per month"

SLA (Service Level Agreement) — A contractual commitment with customers defining expected service levels and consequences of missing them.

• SLA = SLO + consequences (credits, refunds)
• SLA should always be slightly worse than SLO to maintain buffer

Error Budget:

• Error budget = 1 - SLO. If SLO is 99.9%, error budget is 0.1%.
• Once error budget is consumed, freeze feature releases until budget resets.
• Prevents reliability erosion from velocity pressure.

Redundancy Strategies

Strategy	Description	Pros	Cons	Use Case
Active-Active	All instances serve traffic simultaneously	Max throughput, no failover delay	Data consistency complexity	Stateless services, CDN
Active-Passive	One primary, others on standby	Simpler data sync	Waste of standby resources	Databases, legacy systems
N+1	N active + 1 spare	Balanced cost/reliancy	Limited additional capacity	General microservices
N+K	N active + K spares	Higher fault tolerance	More resource waste	Mission-critical systems

Active-Active Design:

• All nodes accept reads and writes
• Conflict resolution needed (last-writer-wins, CRDTs, vector clocks)
• Global load balancer routes to nearest healthy region
• Example: DynamoDB Global Tables, Cassandra multi-DC

Active-Passive Design:

• Primary handles all write traffic
• Replica receives replicated data but does not serve production traffic
• Failover: promote replica to primary, update DNS/load balancer
• Failover time: seconds to minutes depending on detection mechanism
• Example: RDS Multi-AZ, traditional MySQL master-slave

Circuit Breaker Pattern

The circuit breaker prevents cascading failures by short-circuiting calls to a failing dependency.

Three States:

1. Closed (Normal operation)

   • Requests pass through normally
   • Failures are counted
   • When failure threshold is reached → transition to Open

2. Open (Failing fast)

   • All requests are immediately rejected without calling the dependency
   • After a timeout period → transition to Half-Open

3. Half-Open (Probe)

   • A limited number of test requests are allowed through
   • If they succeed → transition to Closed
   • If they fail → transition back to Open

Closed → (failure threshold exceeded) → Open → (timeout expires) → Half-Open
  ↑                                                          |
  └────────────── (probe succeeds) ─────────────────────────┘
  ↑                                                          |
  └────────────── (probe fails) ────────────────────────────┘

Configuration Parameters:

• Failure threshold: number of failures before opening (e.g., 5 failures in 10s)
• Open duration: how long to stay open (e.g., 30 seconds)
• Half-open max calls: number of probe requests (e.g., 3)
• Fallback behavior: return cached data, default response, or error

Implementation: Hystrix (Netflix, now deprecated), Resilience4j, Polly (.NET)

Retry with Exponential Backoff and Jitter

Exponential Backoff:

• Wait time doubles after each retry: 1s, 2s, 4s, 8s, 16s
• Prevents thundering herd by spacing out retry attempts
• Set a maximum retry count (e.g., 5) and maximum delay cap

Jitter:

• Adds randomness to retry delays to prevent synchronized retries
• Types: full jitter, equal jitter, decorrelated jitter

Full Jitter:   sleep = random(0, min(cap, base * 2^attempt))
Equal Jitter:  sleep = min(cap, base * 2^attempt) / 2 + random(0, min(cap, base * 2^attempt) / 2)
Decorrelated:  sleep = min(cap, random(base, previous_sleep * 3))

Best Practices:

• Only retry idempotent operations (GET, PUT, DELETE with idempotency keys)
• Distinguish retryable errors (503, timeout) from non-retryable (400, 401, 403)
• Use circuit breaker alongside retries to avoid hammering a failing service

Bulkhead Pattern

Isolates components so a failure in one does not cascade to others. Named after ship bulkheads that prevent flooding from spreading.

Implementation Strategies:

• Thread pool isolation: Each dependency gets its own thread pool. If Service A's pool is exhausted, Service B's pool is unaffected.
• Semaphore isolation: Limit concurrent calls per dependency using semaphores.
• Connection pool isolation: Separate database connection pools per service.

Service A → [Thread Pool: 10 threads] → Dependency A
Service A → [Thread Pool: 10 threads] → Dependency B
Service A → [Thread Pool: 5 threads]  → Dependency C

Example: Netflix Hystrix bulkhead configuration:

• Each downstream service gets its own command group
• Thread pool size tuned per dependency based on latency and throughput
• When pool is full, requests fail fast with fallback

Chaos Engineering Principles

Chaos engineering proactively introduces failures to discover weaknesses before they cause outages.

Netflix's Chaos Monkey:

• Randomly terminates production instances during business hours
• Forces engineers to build resilient systems that handle instance loss
• Part of the Simian Army: Latency Monkey, Conformity Monkey, Security Monkey

Principles (from "Chaos Engineering" book):

1. Build a hypothesis around steady-state behavior
2. Vary real-world events (network latency, disk failures, process crashes)
3. Run experiments in production (or production-like environments)
4. Automate experiments to run continuously
5. Minimize blast radius

Types of Fault Injection:

• Network: partition, latency, packet loss, DNS failures
• Infrastructure: terminate instances, fill disk, exhaust CPU
• Application: throw exceptions, return errors, corrupt state
• Time: advance clocks, trigger cron jobs early

Tools: Chaos Monkey, Litmus (Kubernetes), Gremlin, AWS Fault Injection Simulator

Payment Processing Reliability Example

A payment system must be highly reliable because failures directly impact revenue and customer trust.

Architecture:

Client → API Gateway → Payment Service → Payment Gateway (Stripe/PayPal)
                              ↓
                         Transaction DB (Primary)
                              ↓
                         Transaction DB (Replica)
                              ↓
                         Message Queue (SQS/Kafka)
                              ↓
                         Notification Service

Reliability Measures:

• Idempotency keys: Every payment request includes a unique idempotency key. Retries don't create duplicate charges.
• Circuit breaker on payment gateway: If Stripe is down, circuit opens after 3 failures. Fallback: queue payment for retry, return "payment pending" to user.
• Retry with backoff: Gateway timeouts retried with exponential backoff (max 3 retries).
• Dual payment providers: Stripe as primary, PayPal as fallback. If Stripe circuit is open, route to PayPal.
• Transactional outbox pattern: Write payment event to DB and message queue in same transaction. Ensures exactly-once processing downstream.
• SLO: 99.99% availability (52 minutes downtime per year), p99 latency < 500ms.
• Active-active: Two regions, each processing independent payment streams with shared DB replication.
• Monitoring: Real-time alerting on error rate > 0.1%, latency p99 > 1s, payment success rate < 99.5%.

Defining Fault Tolerance

Fault tolerance is the ability of a system to continue operating correctly despite the failure of one or more components. A fault-tolerant system degrades gracefully rather than failing completely.

Key Concepts:

• Failure: A component stops working entirely
• Fault: A defect or anomaly in the system (could be partial)
• Error: An incorrect system state resulting from a fault
• Fault Tolerance: System continues operating correctly despite faults

Graceful Degradation

Instead of failing entirely, the system reduces functionality:

• Feature degradation: Disable non-critical features (recommendations, analytics) while keeping core functionality (purchases, search)
• Quality degradation: Serve lower-resolution images, skip personalization
• Capacity degradation: Serve fewer users but maintain quality for those served

Netflix Example: When a dependency fails, Netflix serves cached content instead of personalized recommendations. Users lose recommendations but can still browse and watch.

Amazon Example: If the recommendation engine is down, product pages still load with static content. "Frequently bought together" might be cached or empty rather than blocking page load.

Replication Strategies

Synchronous Replication:

• Write is confirmed only after all replicas acknowledge
• Guarantees zero data loss (RPO = 0)
• Higher write latency
• Used for: financial systems, leader election

Asynchronous Replication:

• Write is confirmed after primary acknowledges
• Replicas may lag behind primary
• Lower write latency but potential data loss
• Used for: read replicas, analytics, most web applications

Semi-synchronous Replication:

• Write confirmed after at least one replica acknowledges
• Balance between durability and latency
• Used for: MySQL semi-sync, Amazon RDS Multi-AZ

Health Checks and Self-Healing

Liveness Probes:

• Check if the application process is running
• If liveness fails → restart the container
• Kubernetes example:

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5

Readiness Probes:

• Check if the application is ready to serve traffic
• If readiness fails → remove from load balancer (don't restart)
• Useful during startup or when temporary overload

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 10

Startup Probes:

• For slow-starting applications
• Prevents liveness probe from killing the app before it starts

startupProbe:
  httpGet:
    path: /health/startup
    port: 8080
  failureThreshold: 30
  periodSeconds: 10

Failure Detection

Heartbeat Mechanism:

• Nodes send periodic heartbeats (e.g., every 5 seconds)
• If no heartbeat received for N intervals → node declared dead
• Too short → false positives; too long → slow detection

Gossip Protocol:

• Each node randomly communicates with a few peers
• Failure information propagates through the cluster
• Used by: Cassandra (Phi Accrual Failure Detector), Consul, Serf
• Eventually consistent but scales well

Phi Accrual Failure Detector:

• Outputs a suspicion level (phi) rather than binary alive/dead
• Phi increases with time since last heartbeat
• Threshold is adaptive based on network conditions
• More nuanced than simple timeout-based detection

Quorum and Consensus

Quorum:

• Minimum number of nodes that must agree for a decision
• In a cluster of N nodes, quorum = N/2 + 1
• Example: 5-node cluster needs 3 nodes for quorum
• Ensures consistency: at most one partition can have quorum

Raft Consensus:

• Leader election: one leader handles all writes
• Log replication: leader replicates log entries to followers
• Safety: committed entries are never lost
• Used by: etcd, Consul, CockroachDB

Paxos:

• More complex but theoretically foundational
• Used by: Google Chubby, Apache ZooKeeper (ZAB variant)

Timeouts and Deadlines

Setting Timeouts:

• Timeout should be based on observed latency distribution (p99 or p99.9)
• Too short → false failures; too long → slow failure detection
• Cascading: total timeout = sum of downstream timeouts (bad) or use deadline propagation (better)

Deadline Propagation:

• Original caller sets an absolute deadline (e.g., "must complete by T+2s")
• Each service checks remaining time before making downstream calls
• Prevents a 2s timeout at each level from becoming 6s end-to-end

Payment System Fault Tolerance

Multi-Layer Fault Tolerance:

1. Transport Layer: TCP connections with keepalive, connection pooling
2. Application Layer: Circuit breakers, retries, bulkheads
3. Data Layer: Replication, backup, WAL (Write-Ahead Log)
4. Infrastructure Layer: Multi-AZ deployment, auto-scaling, health checks
5. Business Layer: Idempotency, compensation transactions (saga pattern)

Saga Pattern for Payment:

Order Service:     CreateOrder → ReserveInventory → ProcessPayment → ConfirmOrder
Failure at Payment: CreateOrder → ReserveInventory → ProcessPayment (FAIL)
Compensation:      CancelOrder ← ReleaseInventory ← RefundPayment ← (triggered)

• Each step has a corresponding compensating transaction
• If any step fails, all previous steps are compensated
• Ensures system returns to a consistent state

RPO and RTO

RPO (Recovery Point Objective):

• Maximum acceptable data loss measured in time
• "We can afford to lose at most 1 hour of data" → RPO = 1 hour
• Determines backup frequency
• RPO = 0 means no data loss (synchronous replication)

RTO (Recovery Time Objective):

• Maximum acceptable time to restore service after a disaster
• "Service must be restored within 4 hours" → RTO = 4 hours
• Determines failover automation level

Strategy	RPO	RTO	Cost
Backup & Restore	Hours	Hours to Days	Low
Pilot Light	Minutes	10-60 minutes	Medium
Warm Standby	Seconds	Minutes	High
Active-Active	0 (near)	Seconds	Very High

Backup and Restore

• RPO: Hours (depends on backup frequency)
• RTO: Hours to days
• Process: Regular backups to secondary region, restore when disaster occurs
• Cheapest option but slowest recovery

Amazon Implementation:

• EBS snapshots to S3 cross-region
• RDS automated backups with point-in-time recovery
• S3 cross-region replication
• DynamoDB point-in-time recovery + on-demand backups

Pilot Light

• RPO: Minutes
• RTO: 10-60 minutes
• Core infrastructure is running in DR region but at minimal capacity
• Data is replicated continuously
• On failover: scale up DR infrastructure to full capacity

Example:

• Primary region: Full application stack
• DR region: Database replica + minimal compute (one small instance)
• On failover: Scale compute, update DNS, promote DB replica

Warm Standby

• RPO: Seconds
• RTO: Minutes
• DR region runs a scaled-down copy of production
• On failover: Scale up to full capacity

Example:

• Primary region: 10 instances
• DR region: 2 instances (20% capacity)
• On failover: Scale DR to 10 instances, redirect traffic

Active-Active (Multi-Region)

• RPO: ~0
• RTO: Seconds (automated)
• Both regions handle production traffic simultaneously
• Global load balancer (Route 53, CloudFront) routes to nearest region
• Data conflicts resolved via CRDTs or last-writer-wins

Amazon Implementation:

• Route 53 latency-based or geolocation routing
• DynamoDB Global Tables (multi-master replication)
• Aurora Global Database (1-second replication lag)
• ElastiCache Global Datastore

Disaster Recovery Testing

Tabletop Exercises:

• Walk through disaster scenarios on paper
• Identify gaps in runbooks and procedures
• Low cost, high value

Game Days:

• Simulate real failures in production-like environment
• Test actual failover procedures
• Measure actual RPO and RTO

Chaos Engineering in DR Context:

• Kill an entire AZ to test cross-AZ failover
• Simulate region failure for multi-region systems
• Verify monitoring and alerting triggers correctly

Multi-Region Failover Architecture

                    ┌─────────────────────┐
                    │   Route 53 (Global)  │
                    │  Latency-based       │
                    │  Routing             │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
    ┌─────────▼─────────┐ ┌───▼───────────┐ ┌──▼──────────────┐
    │   US-EAST-1       │ │  EU-WEST-1    │ │  AP-SOUTHEAST-1 │
    │   (Primary)       │ │  (Secondary)  │ │  (Secondary)    │
    │                   │ │               │ │                 │
    │  ┌─────────────┐  │ │ ┌───────────┐ │ │ ┌─────────────┐ │
    │  │ App Servers │  │ │ │ App       │ │ │ │ App Servers │ │
    │  │ (Auto Scale)│  │ │ │ Servers   │ │ │ │ (Auto Scale)│ │
    │  └──────┬──────┘  │ │ └─────┬─────┘ │ │ └──────┬──────┘ │
    │         │         │ │       │       │ │        │        │
    │  ┌──────▼──────┐  │ │ ┌─────▼─────┐ │ │ ┌──────▼──────┐ │
    │  │ Aurora       │←─┼─┤►│ Aurora     │←─┼─┤►│ Aurora      │ │
    │  │ Primary     │──┼─┼─│ Read      │──┼─┼─│ Read       │ │
    │  │             │  │ │ │ Replica   │ │ │ │ Replica    │ │
    │  └─────────────┘  │ │ └───────────┘ │ │ └────────────┘ │
    └───────────────────┘ └───────────────┘ └────────────────┘

Failover: Route 53 health check fails → remove from DNS → traffic shifts to nearest healthy region → Aurora replica promoted to primary

Failover Procedure:

1. Route 53 health check detects primary region failure
2. DNS record updated to remove primary region (TTL: 60s)
3. Traffic shifts to secondary region within 60 seconds
4. Aurora read replica promoted to primary (automatic or manual)
5. Application continues serving from secondary region
6. When primary recovers: rebuild as replica, then fail back

Amazon-Specific DR Services

Service	DR Feature	RPO	RTO
RDS	Multi-AZ	0	1-2 min
Aurora	Global Database	1 sec	1 min
DynamoDB	Global Tables	~0	seconds
S3	Cross-Region Replication	minutes	N/A
ElastiCache	Global Datastore	seconds	minutes
Route 53	Health checks + failover	N/A	seconds
CloudFront	Origin failover	N/A	seconds
EBS	Cross-region snapshots	hours	hours
EC2	Auto Scaling + Multi-AZ	N/A	minutes