Throughput

Throughput measures how many operations a system can handle per unit time.

Throughput Metrics

Common Metrics:

RPS  = Requests Per Second
TPS  = Transactions Per Second
QPS  = Queries Per Second
MPS  = Messages Per Second
FPS  = Frames Per Second

Conversion:
1000 RPS = 60,000 RPM (requests per minute)
         = 3,600,000 RPH (requests per hour)

Throughput by System Type

System	Typical Throughput
Web Server	1,000-50,000 RPS
API Gateway	10,000-100,000 RPS
Database	1,000-50,000 QPS
Message Queue	10,000-500,000 MPS
Load Balancer	100,000-1,000,000 RPS

Measuring Throughput

Load Testing Setup:

Clients ──── Load Generator ──── System Under Test
   │                                  │
   └── Send N requests/sec            └── Process requests
                                         │
                                    Measure: RPS, latency

Tools:
- Apache Bench (ab)
- wrk
- JMeter
- Locust
- k6

Throughput Calculation

Example: E-commerce System

Target: 10,000 orders/day
Average order time: Business hours (8 hours)
Peak factor: 3x average

Average RPS = 10,000 / (8 × 3600) = 0.35 RPS
Peak RPS = 0.35 × 3 = 1.05 RPS

With safety margin (10x):
Required capacity = 10.5 RPS

Throughput vs Request Size

Small requests (1KB):
- Higher RPS
- Limited by CPU
- Example: API responses

Large requests (1MB):
- Lower RPS
- Limited by network
- Example: File uploads

Throughput (data) = RPS × Average Request Size
1000 RPS × 1KB = 1 MB/s
100 RPS × 1MB = 100 MB/s

A bottleneck is the component that limits overall system throughput.

Common Bottlenecks

System Bottlenecks:

1. CPU Bound
   - High CPU utilization (>80%)
   - Complex computations
   - Solution: Optimize code, add CPUs

2. Memory Bound
   - Low available memory
   - Frequent garbage collection
   - Solution: Add memory, optimize usage

3. Network Bound
   - High network utilization
   - Large payloads
   - Solution: Compression, CDN, optimize payloads

4. Disk Bound
   - High disk I/O
   - Slow storage
   - Solution: SSD, caching, batch I/O

5. Database Bound
   - Slow queries
   - Connection limits
   - Solution: Indexing, connection pooling, read replicas

Identifying Bottlenecks

Monitoring Dashboard:

CPU:     [████████░░] 80%  ← Potential bottleneck
Memory:  [████░░░░░░] 40%  ← OK
Network: [██░░░░░░░░] 20%  ← OK
Disk:    [██████░░░░] 60%  ← OK
DB:      [██████████] 95%  ← Bottleneck!

Action: Optimize database queries first

Bottleneck Analysis

Symptom	Likely Bottleneck	Solution
High CPU	CPU-bound processing	Optimize code, add CPUs
High memory	Memory leak or large datasets	Fix leak, add memory
High network	Large payloads or many requests	Compress, cache, batch
High disk	Frequent I/O operations	Cache, batch writes
High DB latency	Slow queries or connection limits	Index, pool connections

Little's Law

L = λ × W

L = Average number of requests in system
λ = Average arrival rate (throughput)
W = Average time in system (latency)

Example:
λ = 100 RPS
W = 0.5 seconds
L = 100 × 0.5 = 50 requests in system

Implication: To increase throughput, either:
- Reduce latency (W)
- Accept more concurrent requests (L)

Throughput improvement strategies target bottlenecks and optimize resource usage.

Strategy Overview

Improve Throughput
├── Horizontal Scaling
│   ├── Add more servers
│   ├── Load balancing
│   └── Auto-scaling
├── Vertical Scaling
│   ├── More CPU/RAM
│   ├── Faster disks
│   └── Better network
├── Optimization
│   ├── Caching
│   ├── Connection pooling
│   ├── Batch processing
│   └── Async processing
└── Architecture
    ├── Microservices
    ├── Event-driven
    └── Message queues

Horizontal Scaling

Before: 1 server × 1000 RPS = 1000 RPS
After:  10 servers × 1000 RPS = 10,000 RPS

Key: Must be stateless or externalize state

Connection Pooling

Without Pooling:
Request → Create Connection → Use → Close → Response
        (100ms overhead)     (10ms)
Total: 110ms per request

With Pooling:
Request → Get Pooled Connection → Use → Return → Response
        (1ms overhead)           (10ms)
Total: 11ms per request

10x improvement in throughput!

Batch Processing

Individual Processing:
1000 items × 10ms each = 10 seconds

Batch Processing:
10 batches × 100ms each = 1 second

10x improvement by batching!

Tradeoff: Higher latency per item, higher throughput

Message Queues

Synchronous:
Request → Service A → Service B → Service C → Response
        (100ms)       (100ms)       (100ms)     (300ms)

Asynchronous (with queue):
Request → Service A → Queue → Response (100ms)
                  Queue → Service B → Service C

Service A throughput: 3x improvement (doesn't wait)

Throughput Optimization Checklist

1. Identify bottleneck: Profile to find limiting factor
2. Scale horizontally: Add more instances
3. Pool connections: Reuse database connections
4. Cache results: Avoid recomputation
5. Batch operations: Process multiple items together
6. Async non-critical: Don't block on slow operations
7. Optimize queries: Index properly
8. Compress responses: Reduce network transfer