Latency

In distributed systems, latency appears at every layer.

Latency Sources

Full Request Latency:

Client Device     ──── 5ms  (local processing)
     ↓
DNS Resolution    ──── 10ms (domain lookup)
     ↓
TCP Connection     ──── 20ms (3-way handshake)
     ↓
TLS Negotiation    ──── 30ms (if HTTPS)
     ↓
Network Transit    ──── 10-100ms (geographic distance)
     ↓
Server Processing  ──── 50ms (business logic)
     ↓
Database Query     ──── 100ms (data retrieval)
     ↓
Response Transit   ──── 10-100ms (back to client)
     ↓
Client Rendering   ──── 20ms (display response)

Total: ~250-400ms typical

Types of Latency

Type	Description	Typical Value
Network Latency	Time for data to travel between nodes	1-100ms
Processing Latency	Time to process a request	1-50ms
Queue Latency	Time waiting in a queue	0-1000ms
Disk Latency	Time to read/write disk	0.1-10ms
Memory Latency	Time to access RAM	0.0001ms

Geographic Latency

Same Data Center:     0.1-1ms
Same City:           1-5ms
Same Country:        5-20ms
Cross-Continent:     50-200ms

Examples:
New York → London:    70ms
New York → Tokyo:     160ms
New York → Sydney:    200ms

Latency by Component

Component            Latency    Optimization
─────────────────────────────────────────────
Browser rendering    10-50ms    Virtual DOM, SSR
DNS lookup           10-50ms    DNS caching
TCP handshake        20-40ms    Keep-alive, HTTP/2
TLS handshake        30-50ms    TLS session resumption
CDN edge             1-10ms     Edge caching
Application server   10-100ms   Caching, optimization
Database             1-1000ms   Indexing, caching
External API         50-500ms   Async, caching

Latency Percentiles in Practice

Web Application Typical Latency:
P50:  100ms (median user)
P95:  300ms (95% of users)
P99:  500ms (99% of users)
P999: 1000ms (1% of users)

If P99 > 1 second, users start noticing delays.

Accurate latency measurement is essential for identifying bottlenecks and verifying improvements.

Measurement Methods

End-to-End Measurement:

Start Timer ──→ Request ──→ Response ──→ Stop Timer

Total Latency = End Time - Start Time

Server-Side Measurement:
Request Received ──→ Processing ──→ Response Sent
     ↓                                   ↓
  Start Timer                      Stop Timer

Network Measurement:
Ping: ICMP echo request/response
Traceroute: Path and latency per hop

Latency Measurement Tools

Tool	Purpose	What it Measures
curl	HTTP requests	Response time
ping	Network connectivity	Round-trip time
traceroute	Network path	Per-hop latency
ab/wrk	Load testing	Throughput, latency
Prometheus	Monitoring	Application metrics

Measuring Application Latency

# Example: Timing a request
import time

start = time.time()
response = requests.get('https://api.example.com/data')
latency = (time.time() - start) * 1000  # Convert to ms

print(f'Request latency: {latency:.2f}ms')

Latency Distribution

Latency Distribution (1000 requests):

0-50ms:    500 requests (50%)  ████████████
50-100ms:  200 requests (20%)  ████████
100-200ms: 150 requests (15%)  ██████
200-500ms: 100 requests (10%)  ████
500ms+:     50 requests (5%)   ██

P50 = 45ms
P95 = 180ms
P99 = 350ms

Common Measurement Mistakes

1. Measuring only average: Miss slow requests
2. Ignoring cold starts: First request is slower
3. Not warming cache: Cold cache results are misleading
4. Measuring in dev: Production latency differs
5. Not accounting for GC: Garbage collection pauses

Latency reduction strategies target each layer of the request path.

Strategy Overview

Reduce Latency
├── Network
│   ├── CDN (closer to users)
│   ├── HTTP/2 (multiplexing)
│   ├── Keep-alive (connection reuse)
│   └── Compression (smaller payloads)
├── Server
│   ├── Caching (avoid recomputation)
│   ├── Async I/O (non-blocking)
│   ├── Connection pooling (reuse connections)
│   └── Load balancing (closer servers)
├── Database
│   ├── Indexing (faster lookups)
│   ├── Query optimization (less work)
│   ├── Read replicas (closer reads)
│   └── Caching (avoid queries)
└── Client
    ├── Lazy loading (load on demand)
    ├── Prefetching (anticipate needs)
    ├── Code splitting (smaller bundles)
    └── Service workers (offline support)

CDN for Latency Reduction

Without CDN:
User (Tokyo) ──── 160ms ────→ Origin (New York)

With CDN:
User (Tokyo) ──── 5ms ────→ Edge (Tokyo)
Edge (Tokyo) ──── 160ms ────→ Origin (New York)
                             (only on cache miss)

Caching for Latency

Cache Hit Path (fast):
User → App Server → Cache (1ms) → Response

Cache Miss Path (slow):
User → App Server → Cache (miss) → Database (100ms) → Response
                              ↓
                        Cache result for next time

Async Processing

Synchronous (slow):
User → Request → [Wait for DB] → [Wait for Email] → Response
                (200ms)          (300ms)              (500ms total)

Asynchronous (fast):
User → Request → [DB] → Response → [Email job]
                (200ms)  (200ms)

User gets response in 200ms, email sent asynchronously.

Latency Reduction Checklist

1. Profile first: Find where time is actually spent
2. Cache aggressively: Cache everything possible
3. Use CDN: Put content closer to users
4. Optimize queries: Index properly, avoid N+1
5. Async non-critical: Don't block on emails, logs
6. Compress responses: Use gzip/brotli
7. HTTP/2: Multiplex requests
8. Keep connections alive: Avoid TCP handshake overhead