Failure Scenarios

What is a Single Point of Failure (SPOF)?

A SPOF is any component whose failure will bring down the entire system. At Amazon, you are expected to proactively identify and eliminate SPOFs before the interviewer asks. This is a signal that you think like an engineer who builds for production.

Systematic SPOF Checklist

Walk through every layer of your architecture and ask: "If this component dies, what happens?"

Layer	Common SPOFs	How to Eliminate
Load Balancer	Single ALB/NLB instance	Use AWS ALB with multi-AZ, or Route 53 weighted routing
Application Server	Single EC2 instance running the service	Deploy across multiple AZs behind an auto-scaling group with min=2+
Database	Single RDS instance, single table partition	Use RDS Multi-AZ, read replicas, or DynamoDB with multi-AZ replication
Cache	Single Redis/Memcached node	Use ElastiCache with cluster mode and multi-AZ replicas
Storage	Single EBS volume	Use EBS snapshots, S3 with cross-region replication
Network	Single AZ, single internet gateway	Multi-AZ VPC, redundant NAT gateways, multiple AZ subnets
DNS	Single DNS provider	Use Route 53 with health checks and failover routing
Third-party APIs	Single external service dependency	Implement fallbacks, circuit breakers, cached responses
Service Discovery	Single Consul/ZooKeeper node	Run in a 3 or 5 node quorum cluster across AZs
Message Queue	Single SQS queue without DLQ	Add Dead Letter Queue, use SQS with multi-AZ replication

Failure Mode Analysis (FMA)

For every component in your design, answer these questions:

1. What failure modes can occur? (crash, slow response, incorrect data, total unavailability)
2. How do we detect it? (health checks, heartbeats, anomaly detection)
3. What is the blast radius? (does it take down the whole system or just one feature?)
4. What is the recovery path? (automatic failover, manual intervention, restart)
5. What data is at risk? (in-flight requests, cached data, persistent data)

Amazon Interview Pattern

When designing, pause after completing the high-level architecture and say: "Let me walk through the failure scenarios for this design." Then systematically go through each component. This signals production thinking and separates strong candidates from average ones.

Server Crash / Process Failure

Detection: Health checks (HTTP endpoint returning 200, or TCP port check), heartbeats, process monitors.

Handling:

• Auto-scaling group replaces unhealthy instances automatically
• Load balancer drains connections from failed instance before routing new traffic
• Use Spring Boot Actuator /health endpoint or custom health check
• Set health check interval to 10-15 seconds, threshold to 2-3 failures

Database Failure

Detection: Connection pool timeouts, health check queries failing, CloudWatch metrics (CPU, connections, replication lag).

Handling:

• RDS Multi-AZ: automatic failover to standby replica (typically 60-120 seconds)
• DynamoDB: automatic failover, no action needed from application
• For self-managed DBs: sentinel-based failover (Redis), orchestrator (MySQL)
• Application-level: retry logic with exponential backoff, circuit breaker to prevent thundering herd on recovery

Network Partition

Detection: Nodes cannot communicate, split-brain scenarios, heartbeats timing out.

Handling:

• CAP theorem tradeoff: choose AP (availability) over CP (consistency) for most Amazon workloads
• Use consensus algorithms (Raft, Paxos) for coordination services
• DynamoDB handles this automatically with eventual consistency reads
• SQS, S3 are partition-tolerant by design

Cache Failure

Detection: Cache miss rate spike, increased latency on database reads, connection pool exhaustion.

Handling:

• Cache-aside pattern: application checks cache first, falls back to DB on miss
• Cache stampede prevention: singleflight pattern or distributed locks
• ElastiCache automatic failover for Redis cluster mode
• Graceful degradation: serve stale data from cache with "stale-while-revalidate" pattern
• If cache is completely down, circuit breaker trips and application reads directly from DB

DNS Failure

Detection: DNS resolution failures, increased latency from DNS lookups, TTL expiry issues.

Handling:

• Route 53 health checks with failover routing
• Application-level DNS caching with reasonable TTL
• Use IP-based load balancing as fallback
• Multi-provider DNS (Route 53 + CloudFlare as backup)
• Avoid relying on DNS for short-lived connections; prefer service discovery

Circuit Breaker Pattern

States: CLOSED → OPEN → HALF_OPEN

CLOSED: requests flow normally
  → if failure rate exceeds threshold → transition to OPEN

OPEN: all requests fail immediately (fast fail)
  → after timeout period → transition to HALF_OPEN

HALF_OPEN: allow limited requests through
  → if success rate recovers → transition to CLOSED
  → if still failing → transition back to OPEN

Key parameters:

• Failure threshold: 50% of requests in window
• Window size: 10 seconds
• Open duration: 30 seconds
• Half-open requests: 10 requests

Retry with Exponential Backoff

Attempt 1: wait 100ms
Attempt 2: wait 200ms
Attempt 3: wait 400ms
Attempt 4: wait 800ms
Max: 30 seconds

Add jitter: actual_wait = wait * random(0.5, 1.5)
Max retries: 3-5 (depending on operation criticality)

When NOT to retry:

• 4xx client errors (except 429 Too Many Requests)
• Operations that are not idempotent without idempotency keys
• Already retried max times

Graceful Degradation

Not all features are equal. When under stress, degrade non-critical features first:

Priority	Feature	Degradation Strategy
P0	Core transaction	Never degrade, add capacity
P1	Recommendations	Serve cached/stale results
P2	Real-time updates	Switch to polling with longer interval
P3	Analytics	Drop non-critical events, batch later
P4	UI enhancements	Serve static/fallback UI

RPO and RTO

RPO (Recovery Point Objective): How much data can you afford to lose? Measured in time.

• RPO = 0: zero data loss (synchronous replication) — expensive
• RPO = 1 hour: acceptable to lose up to 1 hour of data — common for non-critical systems
• RPO = 1 minute: near-zero loss with asynchronous replication lag under 1 minute

RTO (Recovery Time Objective): How long can the system be down? Measured in time.

• RTO < 1 minute: automatic failover, hot standby — expensive
• RTO < 1 hour: warm standby with automated recovery — moderate cost
• RTO < 24 hours: cold backup with manual restoration — cheapest

Disaster Recovery Strategies

Strategy	RPO	RTO	Cost	Complexity	Use Case
Backup & Restore	Hours	Hours	Low	Low	Dev/test, non-critical
Pilot Light	Minutes	10-30 min	Medium	Medium	Most production workloads
Warm Standby	Seconds	Minutes	High	High	Critical production systems
Hot Standby / Active-Active	Near 0	Near 0	Very High	Very High	Mission-critical (payments, etc.)

Multi-AZ Strategy

What it protects against: Single AZ failure (power, networking, natural disaster in one data center).

Implementation:

• Deploy application across 2-3 AZs with auto-scaling group spanning all AZs
• Use RDS Multi-AZ (synchronous replication to standby)
• Use ElastiCache Multi-AZ with automatic failover
• Load balancer distributes traffic across AZs
• Each AZ has independent power, networking, and cooling

Cost impact: Minimal — AWS charges for data transfer between AZs but not for standby resources in most cases.

Multi-Region Strategy

What it protects against: Entire region failure (rare but possible), regulatory requirements, latency optimization.

Implementation:

• Active-Passive: Primary region serves all traffic, secondary region is standby
  • Route 53 health checks detect region failure
  • Failover to secondary region (DNS change propagation takes 60-120 seconds)
  • Cross-region replication for data (DynamoDB Global Tables, S3 Cross-Region Replication)
• Active-Active: Both regions serve traffic simultaneously
  • Route 53 latency-based or weighted routing
  • Conflict resolution needed for write conflicts (last-write-wins, vector clocks)
  • DynamoDB Global Tables with multi-region replication
  • SQS with cross-region message forwarding

Amazon DR Checklist

When discussing DR in your interview, cover:

1. What is the blast radius? (AZ failure vs. region failure)
2. What is the RPO/RTO target? (align with business requirements)
3. How do we detect a disaster? (health checks, CloudWatch alarms, synthetic monitoring)
4. How do we failover? (automated vs. manual, DNS switch, traffic rerouting)
5. How do we failback? (return to primary after recovery, data sync)
6. How do we test DR? (game days, chaos engineering, regular DR drills)
7. What is the cost? (justify the cost against the business impact of downtime)

Why Data Loss is Catastrophic at Amazon

Amazon handles billions of transactions. Even 0.001% data loss means millions of lost orders, corrupted inventory, or incorrect charges. Data durability is a core design principle.

Replication Strategies

Synchronous Replication:

• Write only acknowledged after all replicas confirm
• Guarantees zero data loss (RPO = 0)
• Higher latency due to round-trip to replicas
• Used for: payment transactions, order placement, inventory updates
• Example: DynamoDB with strong consistency read, RDS Multi-AZ synchronous replication

Asynchronous Replication:

• Write acknowledged after primary confirms, replicas updated later
• Small window of data loss possible (RPO = replication lag)
• Lower latency for writes
• Used for: analytics, logs, non-critical data, read replicas for read scaling
• Example: RDS Read Replicas, DynamoDB Global Tables, S3 Cross-Region Replication

Semi-synchronous Replication:

• Write acknowledged after at least one replica confirms
• Balance between durability and latency
• Used for: MySQL semi-sync, important but not mission-critical data

Write-Ahead Log (WAL)

WAL is a fundamental technique for data durability:

1. Before modifying data, write the intended change to a durable log
2. Apply the change to in-memory structures
3. Periodically flush in-memory changes to persistent storage
4. On crash recovery: replay WAL to restore data to consistent state

Where WAL is used:

• PostgreSQL WAL (Write-Ahead Log)
• MySQL InnoDB redo log
• DynamoDB uses WAL internally for durability
• Kafka commit log (similar concept)
• Raft consensus protocol uses log replication

Backup Strategies

Point-in-Time Recovery (PITR):

• Continuous backup using WAL/binlog
• Can restore to any point in time within retention period
• RDS supports PITR with 1-35 day retention
• DynamoDB supports PITR with 35 day retention

Snapshot-Based Backup:

• Periodic full snapshots of database/storage
• EBS snapshots stored in S3 (incremental)
• RDS automated snapshots daily + manual snapshots
• Schedule: daily snapshots for critical data, hourly for high-value data

Cross-Region Backup:

• S3 Cross-Region Replication (CRR)
• DynamoDB Global Tables for multi-region replication
• Manual snapshot copy to another region for DR

Data Integrity Mechanisms

Checksums:

• Verify data integrity during transfer and at rest
• S3 automatically verifies checksums for all objects
• Use CRC32 or SHA-256 for application-level integrity checks
• DynamoDB checksums on items for consistency verification

Idempotency:

• Ensure operations can be safely retried without side effects
• Use idempotency keys for payment and order operations
• Amazon DynamoDB conditional writes for optimistic concurrency
• SQS deduplication for message processing

Concurrency Control:

• Optimistic locking: version numbers on records, reject writes if version changed
• Pessimistic locking: distributed locks (DynamoDB Lock Client, Redis Redlock)
• DynamoDB transactions for atomic multi-item operations
• Compare-and-set operations for fine-grained concurrency

Amazon-Specific Data Durability

S3 Durability: 99.999999999% (11 nines) — objects are replicated across minimum 3 AZs
DynamoDB Durability: Automatic replication across 3 AZs within a region
EBS Durability: 99.999% availability, snapshots for point-in-time recovery

Anti-Patterns to Avoid

1. No backup strategy: "We'll add backups later" — data loss is irreversible
2. Single-region only: Region failure means total data loss
3. No WAL: Crash during write can corrupt data without WAL
4. Ignoring replication lag: Async replication with high lag = high RPO
5. No idempotency: Retry storms cause duplicate transactions
6. No checksums: Silent data corruption goes undetected