Non-Functional Requirements (HLD)

Understanding Latency Percentiles

Latency is the time it takes for a system to respond to a request. In interviews, always think in percentiles, not averages.

Percentile	Meaning	Example (e.g., API response)
p50 (median)	50% of requests are faster	50ms
p90	90% of requests are faster	120ms
p95	95% of requests are faster	200ms
p99	99% of requests are faster	500ms
p99.9	99.9% of requests are faster	2000ms

Why average is misleading: If you serve 1M requests/day and average latency is 100ms, that hides the fact that 10,000 requests (1%) took over 2 seconds. Users remember bad experiences.

Rule of thumb for interviews:

• Real-time systems (chat, gaming): p99 < 200ms
• Web APIs: p99 < 500ms
• Batch/analytics: p99 < 5s acceptable
• Background jobs: minutes acceptable

Latency Sources

Client ──→ DNS ──→ Load Balancer ──→ App Server ──→ Cache ──→ Database
  │          │           │               │            │          │
  └─Network  └─Resolve   └─Routing       └─Processing └─Hit/Miss └─Query

Each hop adds latency. Identify which component dominates:

• Network latency: Distance between client and server (10-100ms cross-continent)
• Application latency: Business logic processing (1-50ms)
• Database latency: Query execution (1-500ms depending on complexity)
• Cache latency: In-memory lookup (<1ms for local, 1-5ms for distributed)

Throughput vs Latency

These are often confused:

• Latency: Time for one request to complete
• Throughput: Number of requests handled per unit time

A system can have low latency but low throughput (fast but single-threaded) or high latency but high throughput (slow but parallelized, like batch processing).

Amazon's scale example: Amazon.com needs <100ms p99 latency for product pages while handling 100K+ requests/second during peak. This requires caching, CDN, and connection pooling.

Availability Numbers You Must Know

Availability	Downtime/Year	Downtime/Month	Context
99%	3.65 days	7.3 hours	Unacceptable for most production systems
99.9%	8.76 hours	43.8 minutes	Minimum for internal tools
99.99%	52.6 minutes	4.38 minutes	Standard for customer-facing services
99.999%	5.26 minutes	26.3 seconds	Mission-critical (healthcare, finance)
99.9999%	31.5 seconds	2.63 seconds	Telecom-grade

Amazon's target: Most AWS services target 99.99% or higher. An SDE-1 should know that 99.99% means less than 53 minutes of downtime per year.

Availability formula:

Availability = (Total Time - Downtime) / Total Time × 100%

SLA vs SLO vs SLI:

• SLI (Service Level Indicator): What you measure (e.g., p99 latency)
• SLO (Service Level Objective): What you target (e.g., p99 < 200ms)
• SLA (Service Level Agreement): What you promise customers (with penalties)

Horizontal vs Vertical Scaling

Aspect	Vertical (Scale Up)	Horizontal (Scale Out)
Method	Bigger machine	More machines
Limit	Hardware ceiling	Theoretically unlimited
Complexity	Low	High (distributed systems)
Cost	Expensive at scale	Cheaper per unit
Downtime	Required for upgrade	Can be zero-downtime
Example	Upgrade from m5.large to m5.4xlarge	Add more m5.large instances

Interview guidance: Always start with horizontal scaling in your design. Mention vertical as a short-term fix.

Auto-Scaling Strategies

Traffic ↑ → CloudWatch alarm → Add instances → Traffic handled
Traffic ↓ → CloudWatch alarm → Remove instances → Cost saved

Common triggers:

• CPU utilization > 70%
• Request count per target > 1000
• Custom metric (e.g., queue depth)

Scaling policies:

• Target tracking: Maintain metric at target (e.g., CPU at 60%)
• Step scaling: Add instances in steps based on alarm threshold
• Scheduled scaling: Known traffic patterns (e.g., Black Friday)

Multi-AZ and Multi-Region

                    ┌─────────────┐
                    │   Route 53  │
                    │  (DNS)      │
                    └──────┬──────┘
              ┌────────────┴────────────┐
              ▼                         ▼
        ┌───────────┐             ┌───────────┐
        │  Region A │             │  Region B │
        │  (US-East)│             │  (EU-West)│
        └─────┬─────┘             └─────┬─────┘
        ┌─────┴─────┐             ┌─────┴─────┐
        │  AZ-1     │             │  AZ-1     │
        │  AZ-2     │             │  AZ-2     │
        └───────────┘             └───────────┘

Multi-AZ: Protects against data center failure. RDS Multi-AZ, ELB across AZs.
Multi-Region: Protects against region-wide outage. DynamoDB Global Tables, S3 Cross-Region Replication.

Reliability vs Durability

These terms are often confused:

• Reliability: Probability a system performs correctly for a specified time (correctness over time)
• Durability: Probability data is not lost over long periods (data persistence)

Example: A database can be reliable (always returns correct query results) but not durable (loses data on crash). Or durable (data persisted to disk) but not reliable (returns stale data).

CAP Theorem in Practice

The CAP theorem states a distributed system can only guarantee two of three:

• Consistency: Every read receives the most recent write
• Availability: Every request receives a response
• Partition Tolerance: System continues despite network failures

Since network partitions are inevitable, you must choose between CP and AP:

Type	Trade-off	Examples
CP	Sacrifice availability during partition	ZooKeeper, HBase, MongoDB (strong consistency mode)
AP	Sacrifice consistency during partition	Cassandra, DynamoDB, CouchDB

Amazon's approach: Most Amazon services choose AP with eventual consistency because availability is critical for revenue. DynamoDB offers both eventually consistent and strongly consistent reads.

Failure Modes to Consider

1. Single point of failure (SPOF): Remove with redundancy
2. Cascading failures: Circuit breakers, bulkheads
3. Thundering herd: Rate limiting, exponential backoff
4. Data corruption: Checksums, backups, versioning
5. Split brain: Quorum-based decisions, leader election

Redundancy Patterns

Pattern	Protection Against	Implementation
Active-Passive	Server failure	Keep warm standby, failover on failure
Active-Active	Server/DC failure	Multiple active instances, load balanced
N+1	Component failure	N working components + 1 spare
N+M	Multiple failures	N working + M spare (M > 1)

Amazon's 99.99% target requires at minimum Multi-AZ deployment with automated failover.

Security Layers in System Design

┌─────────────────────────────────────────┐
│              Security Layers            │
├─────────────────────────────────────────┤
│ 1. Network Security (VPC, Security Groups) │
│ 2. Authentication (Who are you?)         │
│ 3. Authorization (What can you do?)      │
│ 4. Data Encryption (Protect data)        │
│ 5. Audit Logging (What happened?)        │
│ 6. Compliance (Regulatory requirements)  │
└─────────────────────────────────────────┘

Authentication vs Authorization

Aspect	Authentication	Authorization
Question	Who are you?	What can you do?
Timing	First step	After authentication
Example	Login with username/password	User can view but not delete
Amazon Service	Cognito, IAM	IAM Policies, Resource-based

Common Auth patterns for interviews:

• Session-based: Server stores session, returns cookie (traditional web apps)
• Token-based (JWT): Client stores token, stateless verification (modern APIs)
• OAuth 2.0: Third-party auth ("Sign in with Google")

Encryption

State	Method	Use Case
At Rest	AES-256	Data stored on disk, databases
In Transit	TLS 1.2+	Network communication
In Use	Homomorphic (rare)	Computation on encrypted data

AWS Encryption Services:

• S3: Server-side encryption (SSE-S3, SSE-KMS, SSE-C)
• RDS: Transparent data encryption
• DynamoDB: Encryption at rest (default enabled)
• KMS: Key management for all services

Compliance Considerations

Regulation	Scope	Key Requirement
GDPR	EU users	Right to erasure, data portability
HIPAA	Healthcare data	Encryption, access controls, audit trails
PCI DSS	Payment data	Tokenization, network segmentation
SOX	Financial reporting	Audit trails, access controls

Security in Your Design

In an interview, mention:

1. Input validation: Prevent SQL injection, XSS
2. Rate limiting: Prevent abuse, DDoS
3. Secrets management: Never hardcode, use KMS/Vault
4. Least privilege: Minimal permissions for services/users
5. Audit logging: CloudTrail, application logs

Amazon's Security Principle: "Security is job zero" - always consider security implications in every design decision.

The NFR Prioritization Framework

In a 45-minute interview, you cannot address all NFRs equally. Use this framework:

Step 1: Identify NFRs (5 minutes)
Ask clarifying questions:

• "What are the expected traffic patterns?"
• "What's the acceptable downtime?"
• "Are there regulatory requirements?"
• "What's the data sensitivity level?"

Step 2: Categorize by criticality

Priority	Category	Examples
P0 (Must-have)	Correctness, Basic availability	Data consistency, 99.9% uptime
P1 (Should-have)	Performance, Scalability	p99 latency < 500ms, handle 10x growth
P2 (Nice-to-have)	Advanced features	Multi-region, 99.99% availability
P3 (Future)	Optimization	Cost optimization, advanced monitoring

Step 3: Trade-off decisions

Common trade-offs you'll discuss:

• Consistency vs Availability: Choose based on use case (financial = consistency, social = availability)
• Latency vs Throughput: Batch for throughput, real-time for latency
• Cost vs Performance: Caching improves performance but adds complexity

Amazon-Specific NFR Guidance

Amazon Leadership Principles that influence NFRs:

• Customer Obsession: Latency and availability directly impact customer experience
• Bias for Action: Favor designs that are simple and deployable quickly
• Think Big: Design for scale, but start with MVP
• Dive Deep: Understand the specific NFRs, don't assume

Example: NFR Prioritization for a Chat App

P0 (Must-have):
- Message delivery within 1 second (p99)
- 99.9% availability
- End-to-end encryption for messages

P1 (Should-have):
- Support 100K concurrent users
- Message history for 30 days
- Offline message delivery

P2 (Nice-to-have):
- 99.99% availability
- Multi-region deployment
- Read receipts, typing indicators

P3 (Future):
- Video/voice calls
- File sharing up to 100MB
- Message search

Interview Tips for NFRs

1. Always start with NFRs before diving into the high-level design
2. Quantify when possible: "High availability" means nothing; "99.99% availability" is actionable
3. Explain trade-offs: "We choose eventual consistency here because..."
4. Reference real systems: "Netflix uses Multi-AZ for availability"
5. Know Amazon's scale: If interviewing at Amazon, understand their scale (100K+ microservices, millions of requests/second)