Future Improvements

Scaling Ladder: From Prototype to Global Scale

Different scales require fundamentally different architectures. Here's what changes at each stage.

Stage 1: 0 → 1K Users (Prototype)

Architecture: Single server, monolith, single database

• One EC2 instance or Heroku dyno
• PostgreSQL or MySQL on the same server
• No caching layer yet
• Simple deployment: SSH + git pull

What you need:

• 1-2 CPU cores, 2-4GB RAM
• 10GB storage
• Single region

Focus: Build features, not infrastructure. Speed of iteration matters most.

Stage 2: 1K → 100K Users (Growth)

Architecture: Separate concerns, add caching

• Separate app server from database server
• Add Redis for caching (session, frequent queries)
• Add CDN for static assets
• Load balancer for 2-3 app servers
• Read replicas for database

What changes:

• 3-5 app servers behind a load balancer
• Separate RDS instance with read replicas
• Redis cluster for caching
• CloudFront or Cloudflare CDN

Bottlenecks that emerge:

• Database becomes the bottleneck (vertical scaling limits)
• Session state must be externalized (sticky sessions don't scale)
• Static file serving wastes app server resources

Capacity math:

100K users
- 10% concurrent = 10K concurrent users
- 5% active at any time = 5K requests/sec
- 3 app servers × 2K req/sec each = 6K capacity (33% headroom)

Stage 3: 100K → 1M Users (Scale)

Architecture: Microservices, horizontal scaling

• Decompose monolith into 5-10 services
• Message queue (Kafka/SQS) for async communication
• Database sharding or move to managed service (DynamoDB)
• Dedicated caching layer (ElastiCache)
• Multi-AZ deployment

What changes:

• 10-20 app servers across multiple AZs
• Sharded database or NoSQL
• Kafka for event streaming
• Service mesh for inter-service communication
• CI/CD pipeline with automated testing

Bottlenecks that emerge:

• Single database can't handle write throughput → shard
• Inter-service communication overhead → reduce sync calls
• Deployment complexity → need automated rollouts
• Data consistency across services → saga pattern

Capacity math:

1M users
- 10% concurrent = 100K concurrent users
- 5% active = 50K requests/sec
- 20 app servers × 3K req/sec = 60K capacity
- Database: 5 shards × 10K writes/sec = 50K writes/sec

Stage 4: 1M → 100M Users (Enterprise)

Architecture: Global distribution, multi-region

• Multi-region active-active deployment
• Global database (CockroachDB, Spanner) or regionally sharded
• Edge computing for personalization
• Advanced caching: CDN + application cache + database cache
• Chaos engineering (Chaos Monkey)

What changes:

• 100+ servers across 3+ regions
• Global load balancing (Route 53, Cloudflare)
• Regional data replication with conflict resolution
• Dedicated observability infrastructure
• Feature flags for gradual rollouts

New challenges:

• Cross-region latency (50-200ms)
• Data sovereignty (GDPR, regional data laws)
• Disaster recovery (RTO < 5 minutes, RPO < 1 minute)
• Cost optimization (reserved instances, spot fleet)

Stage 5: 100M → 1B+ Users (Hyperscale)

Architecture: Everything is distributed, everything is eventually consistent

• Thousands of microservices
• Custom database solutions per use case
• ML-driven auto-scaling and capacity prediction
• Zero-trust security model
• Full chaos engineering with game days

What changes:

• Custom hardware (AWS Graviton, custom silicon)
• Proprietary solutions where off-the-shelf fails
• Dedicated SRE teams per service
• Automated incident response
• Cost becomes a primary engineering constraint

Capacity Planning Framework

Capacity planning answers: "Do we have enough infrastructure to handle current and future load?"

Step 1: Measure Current Load

Key metrics to capture:

• Requests per second (RPS) per service
• Data storage size and growth rate
• Bandwidth utilization
• Database connections and query latency
• Cache hit rate

Example baseline:

Current state:
- 10K RPS peak
- 500GB database
- 100MB/s bandwidth
- 200 DB connections
- Cache hit rate: 85%

Step 2: Project Growth

Growth models:

• Linear: +10% per month
• Exponential: 2x every quarter (common for startups)
• Seasonal: 3x during holidays, baseline otherwise
• Event-driven: 10x during a viral moment

Projection formula:

Peak RPS in 12 months = Current Peak × (1 + growth_rate)^12

Example: 10K × (1.15)^12 = 10K × 5.35 = 53.5K RPS

Step 3: Calculate Infrastructure Needs

Compute:

Required instances = Projected RPS / (RPS per instance × target utilization)
Target utilization: 70% (leave 30% headroom for spikes)

53.5K / (3K × 0.7) = 25.5 → 26 app servers

Database:

Projected storage = Current × (1 + growth_rate)^months × retention_multiplier
500GB × 1.15^12 × 1.2 (for indexes/logs) = 3.2TB

Projected writes = 53.5K × 0.3 (write ratio) = 16K writes/sec

Bandwidth:

Projected bandwidth = Current × (1 + growth_rate)^months
100MB/s × 5.35 = 535MB/s

Step 4: Plan for Spikes

Headroom calculation:

• Normal headroom: 30-50% above projected peak
• Spike factor: 2-3x for viral events
• Pre-provisioned capacity vs auto-scaling

Auto-scaling policy:

scaling_policy:
  metric: CPUUtilization
  target: 70%
  scale_out:
    threshold: 75%
    cooldown: 60s
    step: 2 instances
  scale_in:
    threshold: 40%
    cooldown: 300s
    step: 1 instance

Step 5: Cost Analysis

Cost per user:

Total infra cost / Active users = Cost per user
$50K/month / 1M users = $0.05/user/month

Optimization levers:

• Reserved instances (1-3 year): 30-60% savings
• Spot instances: 60-90% savings for fault-tolerant workloads
• Right-sizing: Match instance type to actual usage
• Caching: Reduce database load, which is the most expensive component

Zero-Downtime Database Migrations

Database migrations are one of the riskiest operations in production. Here's how to do them safely.

The Expand-Contract Pattern

Phase 1 — Expand: Add new column/table without breaking existing code

-- Add new column (nullable, no default)
ALTER TABLE users ADD COLUMN email_normalized VARCHAR(255);

Phase 2 — Migrate: Backfill data, update application code to use new column

-- Backfill in batches
UPDATE users SET email_normalized = LOWER(email) WHERE email_normalized IS NULL LIMIT 10000;
-- Repeat until all rows are updated

Phase 3 — Contract: Remove old column after all code is updated

-- Drop old column (only after confirming no code references it)
ALTER TABLE users DROP COLUMN email;

Safe Migration Practices

Backward-compatible migrations:

• Add columns as NULLABLE (no default value that blocks)
• Never rename columns in place — add new, migrate, drop old
• Use feature flags to control which code path uses new schema

Batched backfills:

while True:
    result = db.execute("UPDATE users SET email_normalized = LOWER(email) WHERE email_normalized IS NULL LIMIT 10000")
    if result.rowcount == 0:
        break
    time.sleep(0.1)  # Don't overwhelm the DB

Online schema change tools:

• gh-ost (GitHub): Creates ghost table, applies changes, swaps
• pt-online-schema-change (Percona): Similar approach
• These avoid locking the table during ALTER

Database Migration Checklist

• Write migration as reversible (up and down)
• Test on production-sized dataset (staging)
• Back up database before migration
• Use batched backfills for large tables
• Monitor replication lag during migration
• Have rollback plan ready
• Deploy application code that works with BOTH old and new schema
• Verify application works with new schema
• Remove old schema elements in next deployment

Sharding Migration

When you outgrow a single database, you need to shard. This is a multi-week effort:

Step 1: Choose shard key

• User ID (most common)
• Geographic region
• Time-based (for time-series data)

Step 2: Plan shard distribution

Shard 0: user_id % 1000 [0-999]
Shard 1: user_id % 1000 [1000-1999]
...
Shard 9: user_id % 1000 [9000-9999]

Step 3: Dual-write period

• Write to both old single DB and new sharded DB
• Compare results for consistency
• Read from old DB initially

Step 4: Cutover

• Switch reads to new sharded DB
• Verify consistency
• Stop writes to old DB
• Decommission old DB

Rollback Strategy

Every migration must have a rollback plan:

-- Forward migration
ALTER TABLE users ADD COLUMN phone_verified BOOLEAN DEFAULT FALSE;

-- Rollback migration
ALTER TABLE users DROP COLUMN phone_verified;

For complex migrations (data transformations), you may need to store the original data in a backup table before transforming.

Event Sourcing

Traditional CRUD stores the current state. Event sourcing stores the sequence of events that led to the current state.

Traditional:

accounts table:
| id | balance |
|----|--------|
| 1  | 150     |  (current state only)

Event-sourced:

events table:
| id | aggregate_id | event_type     | data            | timestamp |
|----|-------------|----------------|-----------------|-----------|
| 1  | 1           | AccountCreated | {balance: 100}  | T1        |
| 2  | 1           | MoneyDeposited | {amount: 200}  | T2        |
| 3  | 1           | MoneyWithdrawn | {amount: 150}  | T3        |

Current state: 100 + 200 - 150 = 150 (replayed from events)

Benefits:

• Complete audit trail (required in finance, healthcare)
• Ability to replay events to any point in time
• Debug production issues by replaying event sequence
• Natural fit for domain-driven design

Costs:

• Storage grows indefinitely (need event compaction/snapshotting)
• Querying current state requires replaying events (slow)
• Schema evolution of events is complex

CQRS (Command Query Responsibility Segregation)

CQRS separates write model (commands) from read model (queries):

Commands (writes):          Queries (reads):
- PlaceOrder                - GetOrderStatus
- CancelOrder               - GetOrderHistory
- UpdateShippingAddress     - GetRecentOrders

Architecture:

Client → Command API → Write DB (normalized) → Event Bus → Read DB (denormalized)
Client → Query API  → Read DB (fast reads)

Why separate?

• Write model: Normalized, optimized for consistency
• Read model: Denormalized, optimized for query patterns
• They can use different databases (PostgreSQL for writes, Elasticsearch for reads)

Event Sourcing + CQRS Together

The combination is powerful:

1. Client sends command (PlaceOrder)
2. Command handler validates and emits event (OrderPlaced)
3. Event stored in event store
4. Event bus notifies read model projector
5. Projector updates read-optimized views
6. Client queries read model (fast, denormalized)

Example: E-commerce order

# Command handler
def handle_place_order(command):
    # Validate
    order = Order.create(command.user_id, command.items)
    # Emit event
    event = OrderPlaced(order_id=order.id, user_id=command.user_id, items=command.items)
    event_store.append(event)
    # Return order ID (don't need to query)
    return order.id

# Projector (updates read model)
def on_order_placed(event):
    # Update order summary view
    db.execute("INSERT INTO order_summary VALUES (%s, %s, %s)", 
               event.order_id, event.user_id, len(event.items))
    # Update user's order count
    db.execute("UPDATE user_stats SET order_count = order_count + 1 WHERE user_id = %s",
               event.user_id)

When to Use Event Sourcing + CQRS

Use when:

• Audit trail is required (finance, healthcare, legal)
• Complex domain with many state transitions
• Read and write patterns are very different
• Need to debug production issues by replaying events
• Temporal queries needed ("what was the state at time T?")

Don't use when:

• Simple CRUD application
• Small team (operational complexity is high)
• Strong consistency needed everywhere (eventual consistency is inherent)
• Limited storage budget (events grow indefinitely)

Netflix: From DVD Rental to Global Streaming

Netflix's architecture evolution is a masterclass in scaling through different stages.

Phase 1: DVD Rental (1997-2007)

Architecture: Monolithic, simple

• Single PostgreSQL database
• Monolithic Java application
• Manual warehouse operations
• Simple web frontend

Scale: Millions of DVDs, not millions of concurrent streams

Phase 2: Streaming Launch (2007-2010)

Challenge: Stream video to millions of users simultaneously

Architecture decisions:

• Moved to AWS (early cloud adopter)
• Built content delivery network (Open Connect)
• Started microservices decomposition
• Added Cassandra for user viewing history

Key insight: "If we're going to fail, let's fail doing something we've never done before." — Reed Hastings

Phase 3: Global Expansion (2010-2016)

Challenges:

• 50+ countries, different content libraries
• 100M+ users, billions of events per day
• Sub-second recommendations for个性化 content

Architecture:

• 700+ microservices
• Kafka for event streaming (billions of events/day)
• Cassandra for viewing history (petabytes)
• Elasticsearch for search
• Zuul for API gateway
• Eureka for service discovery

Key decisions:

• Moved from vertical scaling to horizontal scaling
• Adopted chaos engineering (Chaos Monkey) to improve resilience
• Built custom CDN (Open Connect) to reduce AWS costs

Phase 4: Global Scale (2016-Present)

Scale: 200M+ users, 15% of global internet bandwidth

Architecture:

• Multi-region active-active
• Custom database solutions (EVCache for caching)
• ML-driven encoding (per-title encoding optimization)
• Real-time A/B testing framework
• Spinnaker for continuous delivery

Key innovations:

• Per-title encoding: Each title gets optimal encoding settings (saves 20% bandwidth)
• Dynamic CDN: Content placed based on viewing patterns
• Chaos engineering at scale: Regularly kill entire regions to test failover

Lessons for System Design Interviews

1. Start simple: Netflix started with a monolith. Don't over-engineer.
2. Decompose when needed: Microservices came when team size and complexity demanded it.
3. Invest in observability: With 700+ services, monitoring is critical.
4. Embrace failure: Chaos Monkey taught them that failure is inevitable — design for it.
5. Optimize at scale: Per-title encoding saves millions in bandwidth costs.
6. Build vs buy: They built Open Connect (CDN) because commercial solutions couldn't handle their scale.

Scaling Numbers

Metric	2010	2015	2020	2025
Users	20M	70M	200M	300M+
Microservices	~10	~200	~700	~1000+
Daily events	Millions	Billions	Tens of billions	Hundreds of billions
Regions	1	3	6	190+ countries

Always Mention Future Improvements

In system design interviews, ending with future improvements shows you think beyond the immediate problem and understand that systems evolve.

The Closing Statement Template

Always end your design with:

"For the initial design, I focused on [primary requirement] because [reason]. In terms of future improvements, we could:

1. [Scaling]: [How to handle 10x more traffic]
2. [Reliability]: [How to improve availability]
3. [Performance]: [How to reduce latency]
4. [Cost]: [How to reduce infrastructure cost]
5. [Features]: [What new capabilities we could add]"

Example Closing Statements

For a URL shortener:
"For the initial design, I focused on handling 100M URLs with sub-10ms redirects. Future improvements include: adding analytics to track click patterns, implementing geographic routing to serve redirects from the nearest region, and adding link expiration for temporary URLs."

For a chat system:
"For the initial design, I focused on 1-on-1 messaging with delivery guarantees. Future improvements include: adding group chats with membership management, implementing end-to-end encryption, adding voice/video calling via WebRTC, and building a notification service for offline users."

For an e-commerce platform:
"For the initial design, I focused on product catalog and checkout flow. Future improvements include: adding recommendation engine using collaborative filtering, implementing inventory management with supplier integration, adding multi-currency support for international expansion, and building a real-time analytics dashboard for merchants."

Categories of Future Improvements

Category	Examples
Scaling	Sharding, multi-region, CDN, edge computing
Reliability	Chaos engineering, disaster recovery, circuit breakers
Performance	Caching, read replicas, connection pooling, compression
Cost	Reserved instances, spot fleet, storage tiering, deduplication
Features	Analytics, personalization, A/B testing, internationalization
Security	Encryption at rest/transit, audit logging, compliance (SOC2, GDPR)
Observability	Distributed tracing, custom dashboards, anomaly detection

What Interviewers Look For

1. Awareness of limits: You know your design won't scale forever
2. Prioritization: You chose what to build now vs later based on requirements
3. Holistic thinking: You considered reliability, cost, and operations — not just features
4. Growth mindset: You're already thinking about the next iteration

Anti-patterns to Avoid

• Never mention future improvements: Makes you seem like a junior engineer
• Over-focus on future: Spend 80% on current design, 20% on future
• Vague improvements: "We could make it faster" — be specific
• Ignoring current requirements: Don't skip the present to talk about the future