Horizontal Scaling

Horizontal scaling means adding more machines to handle increased load.

How Horizontal Scaling Works

Before (2 servers):
┌─────────────┐     ┌─────────────┐
│  Server 1   │     │  Server 2   │
│  (handling  │     │  (handling  │
│   500 RPS)  │     │   500 RPS)  │
└──────┬──────┘     └──────┬──────┘
       └────────┬──────────┘
                │
         Total: 1000 RPS

After (4 servers):
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Server 1 │ │ Server 2 │ │ Server 3 │ │ Server 4 │
│ (250 RPS)│ │ (250 RPS)│ │ (250 RPS)│ │ (250 RPS)│
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
     └────────────┴────────────┴────────────┘
                      │
               Total: 1000 RPS

Requirements for Horizontal Scaling

1. Stateless Services
   - No session data on server
   - Externalize state (Redis, DB)

2. Load Balancer
   - Distribute traffic
   - Health checks

3. Shared Storage
   - Same data accessible from all servers
   - Database, cache, file storage

4. No Server Affinity
   - Any request can go to any server
   - Sticky sessions (if needed)

Scaling Web Servers

Web Tier Scaling:

1. Add new server
2. Install application
3. Configure to connect to shared resources
4. Add to load balancer pool
5. Verify health checks pass
6. Monitor performance

Auto-scaling:
- CPU > 70% → Add server
- CPU < 30% → Remove server

Scaling Considerations

Factor	Consideration
State	Externalize to Redis/DB
Sessions	Use sticky sessions or external store
Files	Use shared storage (S3, NFS)
Cache	Distributed cache (Redis Cluster)
Database	Read replicas, sharding

Effective load distribution ensures all servers are utilized evenly.

Distribution Strategies

1. Even Distribution
   Each server gets equal traffic
   
   Server 1: 250 RPS
   Server 2: 250 RPS
   Server 3: 250 RPS
   Server 4: 250 RPS

2. Weighted Distribution
   More powerful servers get more traffic
   
   Server 1 (powerful): 400 RPS
   Server 2 (medium): 300 RPS
   Server 3 (small): 200 RPS

3. Capability-Based
   Route based on server capabilities
   
   GPU servers → ML workloads
   CPU servers → General workloads
   Memory servers → In-memory processing

Load Balancer Configuration

upstream web_servers {
    # Round Robin (default)
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
    server 10.0.0.3:8080;

    # Weighted
    server 10.0.0.1:8080 weight=5;  # Powerful
    server 10.0.0.2:8080 weight=3;  # Medium
    server 10.0.0.3:8080 weight=2;  # Small

    # Backup
    server 10.0.0.4:8080 backup;    # Failover
}

Load Distribution Monitoring

Monitor:
- Requests per server
- Response time per server
- Error rate per server
- Connection count per server

Alert if:
- Uneven distribution (>20% variance)
- High error rate on specific server
- Server health check failures

Scaling Database Tier

Database Scaling Options:

1. Read Replicas
   Primary → Replica 1, Replica 2, Replica 3
   Reads go to replicas, writes to primary

2. Connection Pooling
   Servers share pool of DB connections
   Reduce connection overhead

3. Caching Layer
   Redis/Memcached in front of database
   Reduce database load

4. Sharding
   Data split across multiple databases
   Each shard handles subset of data

Managing user sessions in a scaled environment requires careful design.

Session Options

1. Sticky Sessions (Session Affinity)
   Client → Server A (all requests)
   
   Problem: Server failure loses session
   Solution: Backup sessions or external store

2. External Session Store
   Client → Any Server → Redis → Session Data
   
   All servers access same session store

3. Database Session Store
   Client → Any Server → Database → Session Data
   
   Persistent but slower

4. JWT ( Stateless)
   Client → Any Server → Decode Token → Session Data
   
   No server-side storage needed

Sticky Sessions Implementation

# Nginx IP Hash
upstream backend {
    ip_hash;
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
}

# Nginx Cookie-based
upstream backend {
    hash $cookie_jsessionid consistent;
    server 10.0.0.1:8080;
    server 10.0.0.2:8080;
}

External Session Store (Redis)

// Express.js example
const session = require('express-session');
const RedisStore = require('connect-redis').default;
const redis = require('redis');

const client = redis.createClient();

app.use(session({
  store: new RedisStore({ client }),
  secret: 'your-secret',
  resave: false,
  saveUninitialized: false
}));

// Session stored in Redis
// All servers can access same sessions

JWT for Stateless Sessions

1. User logs in
   Server creates JWT with user data

2. JWT returned to client
   Client stores in cookie or header

3. Subsequent requests
   Client sends JWT in Authorization header

4. Server validates JWT
   No database lookup needed

Benefits:
- No server-side storage
- Scalable (any server can handle)
- Fast (no DB lookup)

Tradeoffs:
- Larger request size
- Can't revoke until expiry
- Limited data storage

Session Management Best Practices

1. Externalize session data: Don't store on application server
2. Use appropriate TTL: Sessions should expire
3. Secure session storage: Encrypt sensitive data
4. Monitor session storage: Redis memory usage
5. Plan for failover: Session data must survive server failures