Reliability

Reliability is the probability that a system will perform its intended function without failure over a specified period.

Key Metrics

MTBF (Mean Time Between Failures)
= Total Uptime / Number of Failures

MTTR (Mean Time To Repair)
= Total Repair Time / Number of Repairs

Availability = MTBF / (MTBF + MTTR)

Example Calculation

System Uptime: 364 days
Number of Failures: 4
Total Repair Time: 2 hours (0.083 days)

MTBF = 364 / 4 = 91 days between failures
MTTR = 2 / 4 = 0.5 hours per repair

Availability = 91 / (91 + 0.000057) = 99.99994%

Reliability vs Availability

Aspect	Reliability	Availability
Focus	Correct operation	Uptime
Metric	MTBF, failure rate	Uptime percentage
Question	Does it work correctly?	Is it accessible?
Example	Data never corrupted	System always reachable

Reliability Metrics by System Type

System	MTBF Target	MTTR Target
Web App	30 days	1 hour
Database	90 days	30 minutes
Financial	365 days	5 minutes
Medical	1000+ days	Immediate

Failure Rate

Failure Rate = Number of Failures / Total Operating Time

Example:
1000 hours of operation, 5 failures
Failure Rate = 5 / 1000 = 0.005 failures/hour

Annual Failure Rate = 0.005 × 8760 = 43.8 failures/year

Reliability Engineering Principles

1. Redundancy: Backup components
2. Isolation: Failure containment
3. Graceful Degradation: Reduce functionality under failure
4. Self-healing: Automatic recovery
5. Observability: Monitor and alert

Fault tolerance is the ability of a system to continue operating despite component failures.

Fault Tolerance Patterns

Fault Tolerance
├── Redundancy
│   ├── Active-Active
│   ├── Active-Passive
│   └── N+1 Redundancy
├── Replication
│   ├── Synchronous
│   └── Asynchronous
├── Circuit Breaker
│   ├── Closed (normal)
│   ├── Open (failing)
│   └── Half-Open (testing)
└── Bulkhead
    ├── Separate pools
    └── Isolated failures

Circuit Breaker Pattern

Normal Flow (Closed):
Request → [Circuit Breaker] → Service → Response
                │
                └── Success counter

When failures exceed threshold:
Request → [Circuit Breaker] → ❌ Fail Fast
                │
                └── Open state (no requests to service)

After timeout:
Request → [Circuit Breaker] → Service → Response
                │
                └── Half-Open (test if service recovered)

Bulkhead Pattern

Without Bulkhead:          With Bulkhead:
┌─────────────────┐       ┌─────────────────┐
│   Shared Pool   │       │  Pool A  │ Pool B│
│   (all services)│       │ Service A│ Svc B│
└────────┬────────┘       └────┬─────┴──┬───┘
         │                     │        │
    Service A fails       Svc A fails  Svc B ok
    → Pool exhausted      → Pool A exhausted
    → All services down   → Pool B still works

Graceful Degradation

Full Functionality:
┌─────────────────────────────────┐
│  Full features, all data,      │
│  real-time updates             │
└─────────────────────────────────┘

Degraded Mode (failure):         
┌─────────────────────────────────┐
│  Core features, cached data,   │
│  delayed updates               │
└─────────────────────────────────┘

Minimal Mode (severe failure):   
┌─────────────────────────────────┐
│  Basic read-only access,       │
│  static content only           │
└─────────────────────────────────┘

Fault Tolerance in Practice

Service	Fault Tolerance Strategy
Netflix	Multiple CDNs, regional failover
Google	Geographic redundancy, cell-based architecture
Amazon	Microservices isolation, circuit breakers
Stripe	Idempotent requests, retry with backoff

Redundancy is the foundation of reliable systems. It ensures no single failure causes total system failure.

Levels of Redundancy

Level 1: Component Redundancy
├── Dual power supplies
├── RAID storage
└── Multiple network cards

Level 2: Server Redundancy
├── Active-Active pairs
├── Load balanced clusters
└── Standby replicas

Level 3: Data Redundancy
├── Database replication
├── Cross-region backup
└── Distributed storage

Level 4: Geographic Redundancy
├── Multiple data centers
├── Multi-region deployment
└── CDN distribution

Replication Strategies

Synchronous Replication:
Primary ──── Write ────→ Replica (confirm)
   │                      │
   └── Commit only after replica confirms

Asynchronous Replication:
Primary ──── Write ────→ Replica (fire & forget)
   │                      │
   └── Commit immediately, replica catches up

Semi-synchronous:
Primary ──── Write ────→ At least 1 replica confirms
   │                      │
   └── Commit after 1 replica confirms

Consensus Algorithms

Raft/Paxos:
- Leader election
- Log replication
- Safety guarantees
- Used in: etcd, Consul, ZooKeeper

Quorum:
- Write: W nodes must acknowledge
- Read: R nodes must respond
- W + R > N ensures consistency

Redundancy Tradeoffs

Factor	No Redundancy	Some Redundancy	Full Redundancy
Cost	Low	Medium	High
Complexity	Low	Medium	High
Reliability	Low	Medium	High
Consistency	Strong	Varies	Eventual

Best Practices

1. Automate failover: Don't rely on manual intervention
2. Test redundancy: Regularly simulate failures
3. Monitor replication lag: Ensure replicas are current
4. Use quorum: Balance consistency and availability
5. Document recovery procedures: Know how to recover from each failure