Scaling Strategy

Vertical Scaling (Scale Up)

Vertical scaling means upgrading the existing machine — more CPU, more RAM, faster disk. It is the simplest approach because no application code changes are required.

Advantages:

• No code changes needed — the app runs the same way on a bigger box
• No distributed system complexity (no network partitions, no consensus problems)
• Strong consistency is trivial — single node, single source of truth
• Often the cheapest option for small-to-medium workloads

Disadvantages:

• Hard ceiling — there is a largest machine money can buy
• Single point of failure — if the box dies, everything dies
• Downtime usually required for hardware upgrades
• Cost grows super-linearly at the high end (a 64-core machine costs far more than 2× a 32-core)

When to use vertical scaling:

• Databases that need strong consistency (PostgreSQL, MySQL single-primary)
• Early-stage products where simplicity beats premature optimization
• Workloads with serial bottlenecks that cannot be parallelized

---

Horizontal Scaling (Scale Out)

Horizontal scaling means adding more machines to a pool. The application must be designed to run across multiple instances.

Advantages:

• Virtually unlimited capacity — keep adding nodes
• Fault tolerance — if one node dies, others absorb the load
• Geographic distribution — place nodes near users
• Cost-effective at scale — commodity hardware

Disadvantages:

• Requires stateless services or externalized state (Redis, database)
• Introduces distributed systems challenges: consistency, partitioning, leader election
• Load balancing infrastructure needed
• Deployment and rollback become more complex

When to use horizontal scaling:

• Stateless web services and APIs
• Workloads with predictable high throughput requirements
• Systems requiring high availability (multiple AZs / regions)

---

Decision Matrix

Factor	Vertical	Horizontal
Complexity	Low	High
Max capacity	Bounded by hardware	Nearly unlimited
Fault tolerance	Single point of failure	Inherently redundant
Consistency	Easy (single node)	Requires coordination
Cost at small scale	Lower	Higher (overhead)
Cost at large scale	Prohibitive	Linear

Load Balancer Types

Layer 4 (Transport) Load Balancer:

• Operates at TCP/UDP level
• Routes based on IP address and port number
• Does not inspect HTTP headers or body
• Lower latency, higher throughput
• Examples: AWS NLB, HAProxy (TCP mode)
• Use case: gRPC, database connections, gaming servers

Layer 7 (Application) Load Balancer:

• Operates at HTTP/HTTPS level
• Can route based on URL path, headers, cookies, query strings
• SSL termination offloads encryption work from backends
• Content-based routing: /api/* to API servers, /static/* to CDN
• Examples: AWS ALB, Nginx, Envoy
• Use case: REST APIs, web applications, microservices

---

Load Balancing Algorithms

Round Robin:

• Requests distributed sequentially across all servers
• Assumes all servers have equal capacity
• Works well when servers are homogeneous and requests are similar cost
• Vulnerable if one server is slow — it still gets its equal share

Least Connections:

• Routes to the server with fewest active connections
• Better for long-lived connections (WebSocket, SSE)
• Adapts to uneven request processing times
• Requires tracking connection counts — slight overhead

IP Hash:

• Hash of client IP determines which server receives the request
• Same client always hits same server (session affinity)
• Problem: uneven distribution if IPs are not uniformly distributed (e.g., large corporate NATs)
• Can break if server pool changes (hash ring helps)

Weighted Round Robin:

• Servers assigned weights based on capacity
• A server with weight 3 gets 3× the traffic of weight 1
• Useful when servers have different specs

Least Response Time:

• Routes to server with lowest average response time + fewest connections
• Best for heterogeneous server pools
• Requires continuous monitoring of response times

---

Auto-scaling Strategies

CPU-based scaling:

• Scale out when average CPU > 70% across the fleet
• Scale in when average CPU < 30% for sustained period
• Most common metric, but not always the best (I/O-bound workloads)

Request-count-based scaling:

• Scale based on requests per second (RPS)
• Better for workloads where CPU doesn't correlate with load
• Example: video streaming — low CPU per connection, but many connections

Predictive scaling:

• Uses historical traffic patterns to pre-provision capacity
• Schedule-based: scale up at 8 AM when business hours start, scale down at midnight
• ML-based: AWS Predictive Scaling analyzes 14 days of history
• Eliminates cold-start delays during traffic spikes

Scaling policies:

• Target tracking: maintain CPU at 50% — auto-scaling adjusts capacity
• Step scaling: scale by N instances at threshold T1, by 2N at T2
• Simple scaling: single adjustment with cooldown period

Key consideration — warm-up time:

• New instances take time to boot and become healthy
• Use pre-warming or over-provision during expected spikes
• Set health check grace period to avoid premature termination

Read Replicas

What: Copy of the primary database that accepts read queries only.

How it works:

• Primary handles all writes (INSERT, UPDATE, DELETE)
• Replicas asynchronously replicate from primary via WAL (Write-Ahead Log) or binlog
• Application routes read queries to replicas
• Replication lag is typically milliseconds to seconds

Scaling benefit:

• If primary handles 10K reads/sec, adding 3 replicas gives ~40K reads/sec capacity
• Offloads read-heavy workloads from primary
• Each replica can be in a different AZ for geographic distribution

Trade-offs:

• Replication lag — stale reads possible (eventual consistency)
• Writes still bottlenecked on single primary
• Replica failure reduces read capacity (but doesn't affect writes)
• Complex routing logic needed in application

---

Sharding

What: Splitting a single database into multiple independent shards, each holding a subset of data.

Sharding strategies:

Strategy	How it works	Pros	Cons
Range-based	Shard by value range (e.g., users A-M → shard 1)	Simple, ordered queries efficient	Hotspots if data is skewed
Hash-based	Hash of shard key determines shard	Even distribution	Range queries impossible, resharding hard
Directory-based	Lookup service maps keys to shards	Flexible placement	Extra hop, single point of failure
Geo-based	Shard by user location	Data locality	Uneven shard sizes

Shard key selection (critical):

• Good shard key: high cardinality, even distribution, aligned with access patterns
• Bad shard key: timestamps (all recent data on one shard), status fields (low cardinality)
• Example: user_id for a social network — each user's data on one shard

Cross-shard queries:

• Scatter-gather: query all shards, merge results — slow, expensive
• Denormalization: duplicate data to avoid joins across shards
• Application-level joins: fetch from multiple shards, join in code

---

CQRS (Command Query Responsibility Segregation)

What: Separate the write model (command) from the read model (query).

Architecture:

Write Side → Event Store → Read Side (materialized views)

Benefits:

• Write model optimized for consistency and validation
• Read model optimized for query performance (denormalized views)
• Can scale read and write sides independently
• Each side can use different database technologies

Trade-offs:

• Eventual consistency between write and read sides
• Increased complexity — two models to maintain
• Event sourcing adds durability but increases storage

When to use:

• Read-heavy systems (90%+ reads)
• Complex query patterns that are hard to serve from normalized schema
• Systems where read and write scaling requirements differ dramatically

---

CDN for Static Content

What: Content Delivery Network caches static assets at edge locations worldwide.

What to cache:

• Images, CSS, JavaScript, fonts
• Video files, downloadable resources
• API responses with long TTL (rarely changing data)

Cache invalidation strategies:

• TTL-based: cache expires after N seconds
• Versioned URLs: style.v2.css — new version bypasses cache
• Purge API: manually invalidate specific paths (Cloudflare, CloudFront)
• Content hashing: style.a3b8c9.css — hash in filename

CDN benefits for scaling:

• Reduces origin server load by 80-95%
• Lower latency for users (edge cache hit in < 50ms vs 200ms+ from origin)
• DDoS protection built into most CDN providers
• Cost savings — CDN bandwidth is cheaper than origin bandwidth

Stage 1: 1K Users (MVP)

Architecture: Single server

• Monolithic app on one VM (e.g., t3.medium on AWS)
• PostgreSQL on same server (or RDS db.t3.micro)
• Nginx serves static files and reverse-proxies to app
• No caching layer

Characteristics:

• ~100 concurrent users, ~10 RPS
• Single point of failure acceptable for MVP
• Cost: ~$50-100/month

---

Stage 2: 10K Users

Changes:

• Move database to RDS (managed service, automated backups)
• Add Redis cache (ElastiCache) for frequent queries
• Deploy application behind ALB with 2 instances (horizontal scaling)
• Offload static assets to S3 + CloudFront

Architecture:

Users → CloudFront → ALB → App Server 1
                         → App Server 2
                         → RDS Primary
                         → Redis Cache
                         → S3 (static assets)

Metrics to watch:

• Database connections (max ~100 for db.t3.medium)
• Cache hit ratio (target > 80%)
• Response time P99 (target < 500ms)

---

Stage 3: 100K Users

Changes:

• Auto-scaling group: 4-12 instances based on CPU/request count
• Database read replicas (2-3 replicas for read scaling)
• CDN for all static content + cached API responses
• Background job queue (SQS + workers) for async processing
• Connection pooling (PgBouncer) to manage DB connections

Architecture:

Users → CloudFront → ALB → Auto-scaling group (4-12)
                         → RDS Primary → 2 Read Replicas
                         → ElastiCache Cluster
                         → SQS → Worker Fleet
                         → S3 + CloudFront

New concerns:

• Session affinity or externalized sessions (Redis)
• Database connection limits — use pooling
• Cache stampede protection — singleflight / lock-based refresh

---

Stage 4: 1M Users

Changes:

• Database sharding (split by user_id or region)
• Microservices extraction (auth, billing, notifications as separate services)
• Multi-region deployment for latency + availability
• Event-driven architecture (Kafka for cross-service communication)
• Dedicated search infrastructure (Elasticsearch)
• CDN edge compute (Cloudflare Workers / Lambda@Edge)

Architecture:

Global Users → Route53 (latency-based) → Region A ALB → App Fleet
                                                      → Shard 1 (users A-H)
                                                      → Shard 2 (users I-P)
                                                      → Shard 3 (users Q-Z)
                                                      → Kafka → Analytics Pipeline
                              → Region B ALB → App Fleet (mirror)

Cost optimization:

• Reserved instances for predictable base load
• Spot instances for batch processing
• S3 Intelligent-Tiering for infrequently accessed data
• Right-size instances using CloudWatch metrics

---

Scaling Milestones Summary

Users	Servers	DB Strategy	Cache	Key Cost Driver
1K	1	Single RDS	None	Compute
10K	2-4	RDS + cache	Redis	DB + compute
100K	4-12	Read replicas	Redis cluster	DB replicas + CDN
1M	50+	Sharding	Multi-layer	Multi-region + sharding