Identifying Bottlenecks

What Is a Bottleneck?

A bottleneck is any component that limits overall system throughput. The system can only perform as fast as its slowest component. A single slow database query can make an entire API endpoint slow, regardless of how fast the application server is.

---

Profiling

Application-level profiling:

• CPU profiling: Identify functions consuming the most CPU cycles (flame graphs)
• Memory profiling: Find memory leaks, excessive allocations, garbage collection pressure
• I/O profiling: Detect slow disk reads/writes, excessive file operations
• Tools: pprof (Go), async-profiler (Java), py-spy (Python), clinic.js (Node.js)

Database profiling:

• Slow query log: Log queries exceeding a time threshold (e.g., > 200ms)
• EXPLAIN ANALYZE: Show query execution plan — find sequential scans, missing indexes
• pg_stat_statements (PostgreSQL): Top queries by total time, calls, rows
• Performance Schema (MySQL): Lock waits, table scans, buffer pool hit ratio

Network profiling:

• Latency measurement: traceroute, mtr, ping between services
• Packet capture: Wireshark for protocol-level analysis
• DNS resolution time: Often overlooked — slow DNS adds latency to every request

---

Monitoring

Key metrics to track:

Metric	What it reveals	Tool examples
CPU utilization	Compute bottleneck	CloudWatch, Datadog, Prometheus
Memory usage	Memory leaks, insufficient RAM	CloudWatch, Grafana
Disk I/O (IOPS)	Storage bottleneck	CloudWatch, iostat
Network throughput	Bandwidth saturation	CloudWatch, iftop
Request latency (P50, P95, P99)	Tail latency issues	CloudWatch, New Relic
Error rate	Failing requests	CloudWatch, Sentry
Queue depth	Processing backlog	SQS metrics, Kafka lag
Connection pool usage	DB connection exhaustion	PgBouncer stats, HikariCP metrics

The golden signals (Google SRE):

1. Latency: Time to serve a request
2. Traffic: Requests per second
3. Errors: Rate of failed requests
4. Saturation: How "full" is the resource (CPU, memory, connections)

---

Load Testing

What it reveals:

• Breaking point of the system under realistic load
• Which component degrades first
• Whether auto-scaling kicks in properly

Tools: k6, Locust, Apache JMeter, Artillery

Test types:

• Load test: Simulate expected peak traffic
• Stress test: Push beyond expected peak to find breaking point
• Soak test: Sustained load over hours to find memory leaks, connection exhaustion
• Spike test: Sudden traffic burst to test auto-scaling response

Load testing process:

1. Define realistic user journeys (login → browse → add to cart → checkout)
2. Set baseline metrics (response time, error rate, throughput)
3. Gradually increase load
4. Monitor all four golden signals
5. Identify the first component to degrade — that's your bottleneck
6. Fix it, then repeat

Database Query Bottlenecks

Symptoms: High database CPU, slow query times, connection pool exhaustion

Common causes:

• Missing indexes on frequently queried columns
• N+1 query patterns (1 query to fetch list + N queries for each item)
• Full table scans on large tables
• Lock contention (long-running transactions blocking others)
• Unoptimized JOINs across large tables

Diagnosis:

-- PostgreSQL: Find slow queries
SELECT query, mean_exec_time, calls
FROM pg_stat_statements
ORDER BY mean_exec_time DESC
LIMIT 20;

-- Find missing indexes
SELECT schemaname, relname, seq_scan, seq_tup_read
FROM pg_stat_user_tables
WHERE seq_scan > 100
ORDER BY seq_tup_read DESC;

---

Network Latency

Symptoms: High P99 latency but normal P50, intermittent timeouts

Common causes:

• Cross-region service calls (100-300ms added per hop)
• DNS resolution delays
• TLS handshake overhead (especially for new connections)
• Load balancer added latency
• Insufficient connection pooling (connection setup cost)

Diagnosis:

• Distributed tracing (Jaeger, Zipkin, X-Ray) shows per-hop latency
• Compare P50 vs P99 — large gap suggests tail latency from network
• DNS lookup timing in logs

Real example:
A microservice calls 5 downstream services sequentially. Each adds 50ms network latency. Total: 250ms just from network. Parallelizing these calls reduces it to ~60ms (max of 5 parallel calls).

---

Single Points of Failure (SPOF)

Symptoms: System fails completely when one component goes down

Common SPOFs:

• Single database instance (no replica, no failover)
• Single application server behind DNS
• Single Redis instance for caching
• Single message broker
• Shared filesystem (NFS)

Detection:

• Architecture review: draw the system, identify components with no redundancy
• Chaos engineering: randomly kill components and observe impact
• AWS Well-Architected Tool reviews

---

Memory Bottlenecks

Symptoms: High memory usage, frequent garbage collection, OOM kills

Common causes:

• Memory leaks (objects allocated but never freed)
• Large in-memory data structures (unbounded caches, full result sets)
• Insufficient heap size for JVM applications
• Buffer bloat (excessive buffering in network stacks)

Diagnosis:

• Heap dumps (jmap, VisualVM for Java)
• Memory profiling (Valgrind, heaptrack)
• Container OOM events in Kubernetes logs
• Swap usage — if swap is active, memory is exhausted

---

CPU Bottlenecks

Symptoms: High CPU utilization, slow request processing

Common causes:

• CPU-intensive computation in request path (image processing, encryption)
• Inefficient algorithms (O(n²) where O(n) is possible)
• Excessive garbage collection
• Lock contention causing busy-waiting
• JSON serialization/deserialization overhead

Diagnosis:

• CPU profiling (flame graphs show where CPU time is spent)
• Compare CPU usage across instances — if uneven, load balancing issue
• Thread dumps for Java — find threads in BLOCKED state

---

Connection Pool Exhaustion

Symptoms: Timeout errors, "too many connections" errors, increasing latency

Common causes:

• Connection leaks (opening connections but not closing them)
• Insufficient pool size for traffic volume
• Long-held connections (slow queries holding connections)
• Too many services sharing one pool

Diagnosis:

• Monitor active vs idle connections in pool
• Check for connections in ESTABLISHED state that aren't being used
• Application logs for connection timeout errors

Fix: Connection pooling (PgBouncer for PostgreSQL, HikariCP for Java) with properly tuned pool size. Formula: pool_size = (core_count * 2) + effective_spindle_count (for disk-bound workloads).

Amdahl's Law

Formula: Speedup = 1 / ((1 - P) + P/S)

Where:

• P = fraction of execution time that can be parallelized
• S = speedup factor from parallelization

Key insight: If 50% of your system is serial (P = 0.5), maximum speedup from parallelization is 2×, regardless of how many cores you add. The serial portion dominates.

Implication for bottleneck resolution:

• Optimizing a component that accounts for 10% of latency gives at most 10% improvement
• Optimizing a component that accounts for 90% of latency gives up to 90% improvement
• Always identify the biggest bottleneck first — that's where optimization effort has the most impact

Example:

• API endpoint takes 200ms total
• Database query: 150ms (75%)
• Application logic: 40ms (20%)
• Network serialization: 10ms (5%)
• Optimizing application logic by 50% saves 20ms (10% total improvement)
• Optimizing database query by 50% saves 75ms (37.5% total improvement)

---

Resolution Patterns

**1. Caching

What to cache:

• Computed results that are expensive to generate
• Database query results for read-heavy data
• API responses with predictable access patterns
• Session data, user preferences, configuration

Cache layers:

• Application-level (in-process LRU): < 1ms access, limited by single instance memory
• Distributed cache (Redis/Memcached): 1-5ms access, shared across instances
• CDN cache: 10-50ms access, for static/semi-static content
• Database query cache: built into DB engine, limited by buffer pool size

Cache strategies:

• Cache-aside (lazy loading): App checks cache first, on miss reads from DB and populates cache
• Write-through: Writes go to cache and DB simultaneously
• Write-behind (write-back): Writes go to cache, async flush to DB
• Read-through: Cache handles DB reads internally

Cache pitfalls:

• Cache stampede: many requests hit DB simultaneously when cache expires
• Thundering herd: same issue at larger scale
• Stale data: cache serves outdated information
• Memory pressure: cache evicts useful data

---

**2. Read Replicas

When to use:

• Read-to-write ratio > 10:1
• Database CPU bottleneck from read queries
• Need geographic distribution of reads

Implementation:

• Route reads to replicas via connection string or proxy
• Handle replication lag (read-your-writes consistency via session sticky reads)
• Monitor replica lag and failover if primary fails

---

**3. Connection Pooling

When to use:

• Database connection count hitting limits
• Frequent connection setup/teardown overhead
• Multiple services sharing database connections

Implementation:

• PgBouncer (PostgreSQL): transaction-level pooling for maximum efficiency
• HikariCP (Java): high-performance connection pool
• ProxySQL (MySQL): connection pooling + query routing

Pool sizing: Too few connections → requests wait. Too many → context switching overhead, memory waste. Start with (CPU cores * 2) + disk spindles and adjust based on monitoring.

---

**4. Async Processing

When to use:

• Task doesn't need to complete before responding to user
• Task is slow (email sending, image processing, report generation)
• Task is non-critical (analytics, audit logging)

Implementation:

• Message queue (SQS, Kafka, RabbitMQ) between request handler and worker
• Workers process tasks independently
• User gets immediate response; background workers handle slow operations

Example flow:

Request → App → Queue (immediate response to user)
                     → Worker 1 (process image)
                     → Worker 2 (send email)
                     → Worker 3 (update analytics)

---

**5. Denormalization

When to use:

• Complex JOINs are slow and frequent
• Read-heavy system where writes are less frequent
• Specific query patterns that require data from multiple tables

Trade-off: Write complexity increases (must maintain denormalized data), but read performance improves dramatically.

---

Performance Testing Framework

Load Testing:

• Simulate expected peak traffic (e.g., 10K RPS)
• Measure response time, error rate, resource utilization
• Verify auto-scaling works as expected
• Run for 30-60 minutes to catch gradual degradation

Stress Testing:

• Push beyond expected peak (e.g., 2× expected)
• Find the breaking point — when does the system fail?
• Identify which component fails first
• Test recovery behavior after overload

Soak Testing:

• Sustained load for hours (4-24 hours)
• Catches memory leaks, connection exhaustion, disk fill
• Reveals gradual performance degradation
• Most overlooked test type — critical for production readiness

Capacity Planning:

• Based on load test results, calculate cost per 1K RPS
• Project growth: if traffic doubles in 6 months, plan infrastructure accordingly
• Leave 30-50% headroom for traffic spikes
• Set up alerts for capacity thresholds (80% CPU, 70% memory, etc.)

System Overview

A social media feed service where users can post updates and see a timeline of posts from people they follow. Target: 10M daily active users, 500M feed loads/day.

Initial architecture:

• API gateway → Application servers → PostgreSQL (primary + 1 read replica)
• Redis cache for sessions
• S3 for media storage

---

Identifying the Bottleneck

Load test results at 50K RPS:

• API response time P50: 120ms, P99: 2,800ms
• Error rate: 3.2% (mostly 503 timeouts)
• Database CPU: 98%
• Application CPU: 35%
• Redis hit ratio: 45%

Analysis: The database is clearly the bottleneck. The feed query requires joining the follows table with the posts table, aggregating by recency, and returning 50 posts. With 10M users and millions of follows, this JOIN is extremely expensive.

Amdahl's Law calculation:

• Feed query: 800ms (65% of total latency)
• Application logic: 300ms (25%)
• Network/serialization: 120ms (10%)
• Optimizing application logic by 50% saves 150ms (12% total improvement)
• Optimizing feed query by 50% saves 400ms (32% total improvement)
• Focus on the feed query.

---

Resolution: Fan-out on Write

Problem: The "fan-out on read" approach computes each user's feed on-demand by joining follows + posts. This doesn't scale.

Solution: Fan-out on write — pre-compute each user's feed when a post is created.

Implementation:

Post created → Write to posts table
            → Query user's followers
            → Push post_id to each follower's feed list (Redis sorted set)
            → Feed read: ZREVRANGE on user's feed list (O(log N) operation)

Redis feed structure:

Key: feed:{user_id}
Type: Sorted Set
Score: timestamp
Value: post_id

Feed read:

ZREVRANGE feed:12345 0 49  # Get 50 most recent posts

---

Results After Optimization

• Feed query time: 800ms → 8ms (100× improvement)
• API P99 latency: 2,800ms → 180ms
• Database CPU: 98% → 25%
• Redis memory: increased by 120GB (trade-off)
• Write amplification: each post triggers N writes (one per follower)

---

Remaining Bottlenecks & Further Optimization

New bottleneck: Celebrity problem

• A celebrity with 10M followers — writing one post triggers 10M Redis writes
• Solution: Hybrid approach — fan-out on write for normal users, fan-out on read for celebrities (compute feed on-demand, but cache the result)

New bottleneck: Redis memory at scale

• 10M users × 500 posts in feed × 8 bytes = 40GB minimum
• Solution: Tiered storage — recent 100 posts in Redis, older posts in DynamoDB or Cassandra

New bottleneck: Feed diversity

• Pure chronological feed misses popular content from less-followed accounts
• Solution: Ranking service with ML model that scores posts by relevance, applied after fetching candidate set from Redis

---

Summary Table

Issue	Root Cause	Solution	Impact
Slow feed queries	Expensive JOIN at read time	Fan-out on write (Redis sorted sets)	100× faster reads
Database CPU 98%	Feed queries overwhelming DB	Offload to Redis	CPU drops to 25%
Celebrity write amplification	10M Redis writes per post	Hybrid fan-out (on read for celebrities)	Bounded write cost
Feed diversity	Chronological only	ML ranking layer	Better user engagement
P99 latency 2.8s	Feed query + DB overload	Multi-layer optimization	P99 drops to 180ms