Scalability

Scalability is the ability of a system to handle increased load. There are two fundamental approaches.

Vertical Scaling (Scale Up)

Before:                    After:
┌──────────┐              ┌──────────────────┐
│  4 CPU   │      →       │     16 CPU       │
│  8 GB RAM │              │     64 GB RAM    │
│  100 GB  │              │     1 TB SSD     │
└──────────┘              └──────────────────┘
   Server A                  Bigger Server A

Pros:

• Simple to implement
• No code changes needed
• Strong consistency (single node)
• Lower operational complexity

Cons:

• Hardware limits (max CPU/RAM)
• Single point of failure
• Expensive at high end
• Downtime for upgrades

Horizontal Scaling (Scale Out)

Before:                    After:
┌──────────┐              ┌──────────┐ ┌──────────┐
│  4 CPU   │      →       │  4 CPU   │ │  4 CPU   │
│  8 GB RAM │              │  8 GB RAM │ │  8 GB RAM │
│  100 GB  │              │  100 GB  │ │  100 GB  │
└──────────┘              └──────────┘ └──────────┘
   Server A                  Server A      Server B
                           (same)         (new)

Pros:

• Nearly unlimited scaling
• Fault tolerance (no single point of failure)
• Cost-effective (commodity hardware)
• Zero-downtime scaling

Cons:

• More complex architecture
• Requires stateless services or external state
• Data consistency challenges
• Network overhead between nodes

Comparison

Aspect	Vertical	Horizontal
Complexity	Low	High
Fault Tolerance	Single point of failure	Multiple nodes
Maximum Scale	Hardware limited	Nearly unlimited
Cost	Expensive at scale	Commodity hardware
Downtime	Required for upgrades	Zero-downtime
Consistency	Strong (single node)	Eventual (distributed)

When to Use Each

Vertical Scaling:

• Databases (easier than sharding)
• Small to medium workloads
• When simplicity is priority
• Legacy applications

Horizontal Scaling:

• Web servers behind load balancer
• Microservices architectures
• Systems requiring high availability
• Rapidly growing systems

A load balancer distributes incoming requests across multiple servers.

Basic Load Balancing

                    ┌─────────────┐
                    │    Client    │
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │Load Balancer│
                    └──────┬──────
               ┌───────────┼───────────┐
               │           │           │
        ┌──────▼──┐  ┌─────▼───┐  ┌────▼─────┐
        │Server 1 │  │Server 2 │  │Server 3  │
        └─────────┘  └─────────┘  └──────────┘

Load Balancing Algorithms

Algorithm	How it Works	Best For
Round Robin	Sequential distribution	Equal-capacity servers
Weighted Round Robin	Proportional to weight	Mixed-capacity servers
Least Connections	Fewest active connections	Long-lived connections
IP Hash	Same client → same server	Session affinity needed
Random	Random selection	Simple, no state

Round Robin Example

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A (back to start)
Request 5 → Server B
...

Least Connections Example

Server A: 5 active connections
Server B: 2 active connections  ← Next request goes here
Server C: 8 active connections

New Request → Server B (fewest connections)

Health Checks

Load Balancer periodically checks:

GET /health → 200 OK → Server is healthy ✓
GET /health → timeout → Server is down ✗

If server fails health check:
- Remove from rotation
- Stop sending traffic
- Retry health check later
- Re-add when healthy

Layer 4 vs Layer 7 Load Balancing

Aspect	Layer 4	Layer 7
Info Used	IP + Port	Full HTTP request
Speed	Faster	Slower
Features	Basic routing	Content-based routing
SSL	Passthrough	Termination
Example	AWS NLB	AWS ALB, Nginx

Auto-scaling automatically adjusts the number of servers based on demand.

How Auto-Scaling Works

Metric (CPU, requests/sec)
        │
        ▼
┌─────────────┐
│  Scaling    │
│  Policy     │
└──────┬──────
       │
       ├── Above threshold → Scale Out (add servers)
       └── Below threshold → Scale In (remove servers)

Scaling Policies

Policy	Trigger	Action
Target Tracking	Metric hits target	Maintain target
Step Scaling	Metric crosses thresholds	Add/remove instances in steps
Scheduled	Time-based	Scale at specific times
Predictive	ML-based prediction	Pre-scale for expected load

Auto-Scaling Configuration

# Example: AWS Auto Scaling Group
AutoScalingGroup:
  MinSize: 2
  MaxSize: 10
  DesiredCapacity: 4
  ScalingPolicy:
    TargetValue: 70.0  # CPU utilization
    MetricType: AverageCPUUtilization
    ScaleOut:
      Increase: 2 instances
      Cooldown: 300 seconds
    ScaleIn:
      Decrease: 1 instance
      Cooldown: 600 seconds

Scaling Metrics

Common Metrics for Scaling Decisions:

1. CPU Utilization
   - Scale out when > 70%
   - Scale in when < 30%

2. Request Count per Target
   - Scale out when > 1000 RPS per instance

3. Memory Utilization
   - Scale out when > 80%

4. Queue Depth
   - Scale out when > 100 messages per worker

5. Custom Metrics
   - Business-specific: orders/min, search queries/sec

Auto-Scaling Best Practices

1. Set minimum capacity: Always have at least 2 instances for availability
2. Set maximum capacity: Prevent runaway scaling costs
3. Use cooldown periods: Avoid thrashing (rapid scale in/out)
4. Monitor scaling events: Alert on unusual scaling patterns
5. Test scaling: Regularly verify scaling works as expected

Scaling Challenge: Cold Start

Problem: New instance takes time to start

Solution: Keep warm instances
- Minimum capacity > 0
- Pre-warming instances
- Using reserved capacity for baseline
``