Performance

Latency and throughput are two fundamental performance metrics that often involve tradeoffs.

Definitions

Latency = Time to complete a single operation
         (How fast is one request?)

Throughput = Number of operations per unit time
            (How many requests per second?)

Analogy: Highway

Latency = Time for one car to travel the highway
Throughput = Number of cars passing a point per hour

Latency:  ──→──→──→──→──→  (30 minutes per car)
Throughput: ════════════════  (1000 cars/hour)

You can have:
- Low latency, low throughput (sports car on empty road)
- High latency, high throughput (traffic jam, many cars)
- Low latency, high throughput (highway with many lanes)

Latency vs Throughput Tradeoffs

Scenario	Latency	Throughput	Example
Batch processing	High	High	Data pipeline
Real-time processing	Low	Low	Video call
Web serving	Low	High	E-commerce
Analytics	High	High	Report generation

The Tradeoff

Optimizing for Latency:
- Process each request immediately
- May use more resources per request
- Better for real-time systems

Optimizing for Throughput:
- Batch requests together
- Amortize overhead across many requests
- Better for batch processing

Example:
1000 requests to process

Option A (Latency optimized):
- Process each immediately
- Latency: 10ms per request
- Total time: 10 seconds
- Throughput: 100 req/sec

Option B (Throughput optimized):
- Batch 100 requests together
- Latency: 100ms per batch
- Total time: 1 second
- Throughput: 1000 req/sec

When to Optimize for What

System Type	Optimize For	Why
Chat app	Latency	Real-time conversation
Video streaming	Throughput	Large data transfer
E-commerce	Both	Fast page loads, many users
Data pipeline	Throughput	Process large datasets
Gaming	Latency	Real-time interaction

Understanding performance metrics helps you design and optimize systems effectively.

Latency Metrics

Response Time Breakdown:

Client ──→ DNS ──→ TCP ──→ TLS ──→ Server ──→ DB ──→ Response
10ms     5ms    20ms    30ms    50ms      100ms   5ms

Total: 220ms

Percentiles:
- P50 (median): 150ms (50% of requests)
- P95: 300ms (95% of requests)
- P99: 500ms (99% of requests)
- P999: 1000ms (99.9% of requests)

Why Percentiles Matter

Average vs Percentile:

99 requests: 100ms each
1 request: 5000ms (slow)

Average: (99 × 100 + 5000) / 100 = 149ms
P99: 5000ms

The average looks good, but 1% of users experience 50x slower!

Throughput Metrics

Metric	Definition	Example
RPS	Requests per second	10,000 RPS
TPS	Transactions per second	1,000 TPS
QPS	Queries per second	50,000 QPS
Bandwidth	Data per second	1 Gbps

Performance Metrics by Layer

Application Layer:
- Response time (P50, P95, P99)
- Error rate
- Request rate

Database Layer:
- Query latency
- Connection count
- Cache hit rate

Infrastructure Layer:
- CPU utilization
- Memory usage
- Disk I/O
- Network I/O

SLI, SLO, SLA

SLI (Service Level Indicator):
- Metric: P99 latency
- Value: 250ms

SLO (Service Level Objective):
- Target: P99 latency < 300ms
- Window: 30 days

SLA (Service Level Agreement):
- Contract: 99.9% availability
- Penalty: Credit if missed

Performance optimization is about identifying and removing bottlenecks.

Optimization Process

1. Measure current performance
2. Identify bottleneck
3. Optimize bottleneck
4. Measure improvement
5. Repeat until target met

Common Bottlenecks:
├── Network: High latency, low bandwidth
├── CPU: High utilization, slow computation
├── Memory: Low available memory, GC pauses
├── Disk: Slow I/O, high latency
└── Database: Slow queries, connection limits

Optimization Strategies

Layer	Strategy	Impact
Client	Caching, compression, lazy loading	Reduces network
Network	CDN, HTTP/2, keep-alive	Reduces latency
Server	Connection pooling, async I/O	Increases throughput
Database	Indexing, query optimization	Reduces query time
Application	Algorithm optimization, caching	Reduces computation

Caching Strategy

Cache Hierarchy:

Browser Cache → CDN → Application Cache → Database Cache
    ↓            ↓          ↓               ↓
   0ms        10ms       20ms            100ms

Cache Hit: Request served from cache (fast)
Cache Miss: Request goes to origin (slow)

Cache Strategy:
- Read-Through: App reads cache, falls back to DB
- Write-Through: Write to cache and DB simultaneously
- Write-Behind: Write to cache, async write to DB

Database Optimization

-- Before: Full table scan
SELECT * FROM orders WHERE customer_id = 123;
-- Time: 500ms (scans 1M rows)

-- After: Add index
CREATE INDEX idx_customer ON orders(customer_id);
SELECT * FROM orders WHERE customer_id = 123;
-- Time: 5ms (index lookup)

Performance Testing

Load Testing:
- Normal load: 1000 RPS
- Expected peak: 5000 RPS
- Stress test: 10000 RPS

Metrics to Monitor:
- Response time (P50, P95, P99)
- Throughput (RPS)
- Error rate
- CPU/Memory usage

Performance Budgets

Page Load Budget:
- Total: < 3 seconds
- DNS: < 50ms
- TCP: < 100ms
- TLS: < 100ms
- TTFB: < 200ms
- Download: < 1000ms
- Render: < 500ms