Improving the Design

Why Iteration Matters in Interviews

Amazon interviewers do not want to see a perfect final architecture. They want to see how you think about evolving a system over time. A design that works for 1K users is very different from one that works for 100M users. Showing this evolution demonstrates:

1. Pragmatism: You do not over-engineer from day one.
2. Growth thinking: You can anticipate future needs.
3. Tradeoff awareness: You know what to add and when.

The 3-Phase Iteration Framework

Every system design should have at least 3 phases:

Phase	Scale	Characteristics	Architecture
V1: MVP	1K-10K users	Single server, simple DB, no caching	Monolith, single database, direct connections
V2: Growth	10K-1M users	Caching, read replicas, basic CDN	Stateless services, Redis cache, CDN for static
V3: Scale	1M-100M+ users	Sharding, microservices, queues, global	Distributed, event-driven, multi-region

How to Present Iteration in an Interview

Do not just describe V3. Walk the interviewer through the evolution:

Let me start with a simple V1. For our URL shortener, V1 is a single Flask app with a PostgreSQL database. Short codes are generated using a counter. This handles about 1K requests per second easily.

When we hit 10K RPM, the database read path becomes the bottleneck. So in V2, we add a Redis cache with a cache-aside pattern. We also add a CDN for serving the redirect since redirects are read-heavy and do not change, CDN gives us massive throughput. We also move to a stateless API layer behind an ALB so we can scale horizontally.

At 100K+ RPM, we need to shard the database by short_code prefix. We also introduce async processing for analytics events via SQS. And we add read replicas for the database.

The When-to-Add Decision Framework

Do not add components because they are cool. Add them because they solve a specific problem:

Problem	Solution	When to Add
Database reads are slow	Cache layer (Redis)	When DB read latency > 50ms or read load is high
Static assets are slow globally	CDN	When users are geographically distributed
Write throughput is limited	Database sharding	When single DB cannot handle write load
Service A waits for Service B	Message queue (SQS/Kafka)	When services can be async or need decoupling
Single point of failure	Redundancy, replication	When downtime has direct customer impact
Deployments cause downtime	Blue-green, canary	When you need zero-downtime deploys
Hard to scale independently	Microservices decomposition	When parts of the system scale differently

Common Anti-Patterns

Anti-Pattern 1: Over-engineering V1.

• Bad: We will use Kafka, Cassandra, and Kubernetes from day one.
• Good: V1 is a simple monolith with PostgreSQL. We add complexity as we hit limits.

Anti-Pattern 2: Under-engineering at scale.

• Bad: We will keep the same architecture forever.
• Good: At 1M users, we need to shard the database and add caching.

Anti-Pattern 3: Adding everything at once.

• Bad: We add caching, queues, CDN, and microservices in V2.
• Good: V2 adds caching and CDN. V3 adds queues and sharding.

The Why-Now Justification

For each improvement, explain why it is needed at this scale, not before:

• Caching: We add Redis at 10K RPM because the database can handle about 5K reads/sec on a single instance. Beyond that, we need caching.
• CDN: We add CloudFront when users spread across regions. Before that, a single region serves everyone fine.
• Queues: We add SQS when we need to decouple services or handle traffic spikes. In V1, synchronous calls are simpler and sufficient.

Feature Evolution Roadmap

Amazon interviewers love seeing how you would add features over time. Here is how to present feature additions strategically.

Real-Time Updates

V1: Polling

• Client polls the server every 30 seconds for updates.
• Simple but wasteful since most polls return no new data.

V2: Long Polling

• Client sends a request, server holds it until data is available or timeout.
• Reduces wasted requests but still uses HTTP.

V3: WebSockets

• Persistent bidirectional connection.
• Real-time push for chat, notifications, live feeds.
• Requires sticky sessions or a pub/sub layer (Redis Pub/Sub, SNS).

V4: Server-Sent Events (SSE)

• One-way real-time from server to client.
• Simpler than WebSockets for read-only updates (news feeds, dashboards).

Analytics and Monitoring

V1: Synchronous Logging

• Write analytics events directly to the database.
• Simple but adds latency to user requests.

V2: Async Event Stream

• Fire events to an SQS/Kafka queue.
• Worker processes consume and write to an analytics store (Redshift, ClickHouse).
• User requests are not blocked.

V3: Real-Time Analytics

• Stream events to Kinesis/Kafka.
• Flink/Spark jobs process in real-time.
• Dashboard shows live metrics.

Personalization

V1: Rule-Based

• Simple rules like Users who bought X also bought Y.
• Hardcoded in the application.

V2: Collaborative Filtering

• ML model trained offline on user behavior.
• Predictions served from a feature store.

V3: Real-Time Personalization

• Stream user events to ML pipeline.
• Model updates in near real-time.
• Personalized results served from cache.

Notifications

V1: In-App Only

• User must open the app to see notifications.
• Simple, no infrastructure needed.

V2: Push Notifications

• APNs (iOS) and FCM (Android) integration.
• Need a notification service that manages device tokens and sends to push providers.

V3: Multi-Channel

• Push + Email + SMS.
• User preferences for channel selection.
• Need a notification orchestrator that routes based on user settings.

Search

V1: Database LIKE Query

• Simple but slow and does not scale.

V2: Full-Text Search

• PostgreSQL full-text search or Elasticsearch.
• Better performance, supports stemming, ranking.

V3: Advanced Search

• Elasticsearch with analyzers, synonyms, fuzzy matching.
• Autocomplete, faceted search, personalized results.

Feature Addition Timing Table

Feature	V1 (1K users)	V2 (100K users)	V3 (10M users)
Real-time	Polling	Long polling	WebSockets + Pub/Sub
Analytics	Sync logging	Async (SQS)	Real-time (Kafka + Flink)
Personalization	Rule-based	Offline ML	Real-time ML
Notifications	In-app	Push	Multi-channel
Search	DB LIKE	Elasticsearch	Advanced search + autocomplete
Auth	Session-based	JWT + OAuth	Federation + MFA
Caching	None	Redis	Multi-tier (CDN + Redis + local)

The Scaling Hierarchy

When scaling a system, improvements should be applied in order of impact and ease:

Order	Improvement	Impact	Effort	When to Add
1	Caching	High	Low	First bottleneck hit
2	CDN	High	Low	Global users or static content
3	Read Replicas	High	Medium	Database read bottleneck
4	Message Queue	High	Medium	Need async processing or decoupling
5	Database Sharding	Very High	High	Write bottleneck or data too large
6	Microservices	Medium	Very High	Different parts scale differently
7	Multi-Region	Very High	Very High	Global latency or disaster recovery

Caching Layer

When: Database reads are the bottleneck.

Architecture:

Client -> API Gateway -> Service -> Redis Cache -> Database
                                  (hit = return)
                                  (miss = query DB, populate cache)

Key decisions:

• Cache-aside vs Write-through: Cache-aside is simpler and more common. Write-through adds complexity but ensures freshness.
• TTL: Set based on data freshness requirements. 5 min for product pages, 1 min for inventory counts.
• Eviction policy: LRU for most use cases. LFU if access patterns are stable.
• Stampede prevention: Use distributed locks (Redis SETNX) or probabilistic early expiration.

Impact: Reduces database reads by 80-95%. A single Redis instance can handle 100K+ reads/sec.

CDN (Content Delivery Network)

When: Users are geographically distributed or you have static/semi-static content.

Architecture:

User -> CloudFront Edge -> Origin (S3 or ALB)
      (cached = fast)
      (miss = fetch from origin, cache at edge)

What to cache:

• Static assets (images, CSS, JS) cache aggressively (24h+)
• API responses for read-heavy endpoints cache with shorter TTL (5-60 min)
• Product pages cache for seconds to minutes depending on inventory sensitivity

Impact: Reduces latency by 50-80% for global users. Offloads 60-90% of traffic from origin.

Database Read Replicas

When: Read throughput exceeds single database capacity.

Architecture:

Service -> Primary DB (writes)
       -> Replica 1 (reads)
       -> Replica 2 (reads)
       -> Replica 3 (reads)

Key decisions:

• Synchronous vs async replication: Async is standard (eventual consistency, ~100ms lag). Sync is rare (strong consistency, higher latency).
• Read routing: Application-level routing or proxy (ProxySQL, RDS Proxy).
• Replica lag monitoring: Alert if lag > 1 second.

Impact: Linear read scaling. Each replica adds ~10K reads/sec capacity.

Message Queue

When: Services need to be decoupled, or you need to handle traffic spikes.

Architecture:

Service A -> SQS/Kafka -> Service B
           (buffer)
           (processes at own pace)

Use cases:

• Order processing: User places order, order service writes to queue, payment service processes async.
• Analytics: Fire events to queue, analytics worker consumes and writes to data warehouse.
• Email notifications: Trigger email, queue, email worker sends via SES.

Key decisions:

• SQS vs Kafka: SQS is simpler, managed, good for most use cases. Kafka for high throughput, event sourcing, stream processing.
• Dead letter queue: Always configure for failed messages.
• Visibility timeout: Set based on processing time + buffer.

Impact: Decouples services, handles traffic spikes, improves reliability.

Database Sharding

When: Single database cannot handle write load or data volume.

Architecture:

Service -> Shard Router -> Shard 1 (users A-M)
                      -> Shard 2 (users N-Z)
                      -> Shard 3 (users 0-9)

Sharding strategies:

• Hash-based: Consistent hash on a key (e.g., user_id). Even distribution but range queries are hard.
• Range-based: Shard by key range (e.g., user_id 1-1M goes to shard 1). Good for range queries but can create hot spots.
• Directory-based: Lookup table maps keys to shards. Flexible but adds a lookup hop.

Challenges:

• Cross-shard joins are expensive.
• Rebalancing when adding shards.
• Global secondary indexes need careful design.

Impact: Horizontal write scaling. Each shard adds capacity. Data stays manageable.

Microservices Decomposition

When: Different parts of the system scale differently or teams need independence.

Architecture:

API Gateway -> User Service
            -> Order Service
            -> Product Service
            -> Payment Service
            -> Notification Service

When to split:

• Different scaling profiles (e.g., search scales differently than checkout).
• Different deployment cycles (e.g., payment service updates weekly, product service daily).
• Team boundaries (Amazon two-pizza team rule).

Challenges:

• Distributed transactions are hard.
• Network latency between services.
• Operational complexity (more services to monitor, deploy, debug).

Impact: Independent scaling, independent deployment, team autonomy. But adds significant complexity.

Scaling Decision Matrix

Current Limit	First Improvement	Second Improvement	Third Improvement
DB read latency	Redis cache	Read replicas	CDN
DB write throughput	Write optimization (batching)	Sharding	Event sourcing
Global latency	CDN	Multi-region	Edge computing
Traffic spikes	Auto-scaling + queues	Load shedding	Pre-provisioning
Deployment risk	Blue-green deploys	Feature flags	Canary releases
Team velocity	Microservices	Shared libraries	API contracts

The V1, V2, V3 Narrative

Amazon interviewers want to see that you can evolve a design over time. The best way to show this is the versioned narrative:

In V1, we do X. In V2, when we hit Y constraint, we add Z. At scale W, we switch to approach A.

This demonstrates:

• You do not over-engineer.
• You understand scaling constraints.
• You know what to add and when.

How to Structure the Growth Narrative

Step 1: Start Simple (V1)

• Single server, single database, no caching.
• Explain why this is sufficient for early scale.
• This handles 1K RPM easily and is simple to operate.

Step 2: Add When Needed (V2)

• Caching, read replicas, CDN, stateless services.
• Explain the trigger: When we hit 10K RPM, the database cannot keep up.
• We add Redis for caching and move to stateless services behind an ALB.

Step 3: Scale Aggressively (V3)

• Sharding, microservices, queues, multi-region.
• Explain the trigger: At 1M RPM, we need horizontal scaling at every layer.
• We shard the database by user_id, add Kafka for event processing, and deploy across regions.

Concrete Example: URL Shortener

V1: Simple Monolith

Flask App -> PostgreSQL
Short code: counter-based (1000 -> abc)
Handles: 1K RPM

V2: Add Caching + CDN

Flask App -> Redis (cache-aside) -> PostgreSQL
CloudFront for redirects
Stateless Flask behind ALB
Handles: 10K RPM

V3: Shard + Async

Flask App -> Redis -> Sharded PostgreSQL (by short_code prefix)
SQS for analytics events
Multiple regions with DNS routing
Handles: 100K+ RPM

Concrete Example: Chat System

V1: Simple Polling

Client polls server every 30s
Messages stored in PostgreSQL
Handles: 100 users in a single room

V2: WebSockets + Redis

WebSocket connections to server
Redis Pub/Sub for fan-out
Messages stored in Cassandra (append-only)
Handles: 10K concurrent users

V3: Global Chat

WebSocket connections to nearest region
Redis cluster for Pub/Sub
Cassandra with multi-region replication
Handles: 1M+ concurrent users globally

Common Improvements to Mention

These are safe, high-impact improvements to add during iteration:

1. Add a CDN Low effort, high impact for global users.
2. Add a cache layer Low-medium effort, high impact for read-heavy systems.
3. Introduce a message queue Medium effort, high impact for async processing.
4. Add database read replicas Medium effort, high impact for read scaling.
5. Implement feature flags Low effort, high impact for safe deployments.
6. Add distributed tracing Low effort, high impact for debugging.
7. Implement circuit breakers Low effort, high impact for resilience.
8. Add rate limiting Low effort, high impact for abuse prevention.

What NOT to Add

• Do not add microservices in V1. Start monolith, split when team or scaling demands it.
• Do not add Kubernetes in V1. Start with EC2/ECS, migrate when operational complexity demands it.
• Do not add Kafka in V1. Start with SQS, migrate when throughput or event sourcing needs it.
• Do not add Cassandra in V1. Start with PostgreSQL, migrate when data model or scale demands it.

Amazon Growth Expectations

Amazon specifically looks for:

• Start with the customer and work backwards. Your V1 should be simple enough to ship quickly.
• Bias for action. Do not spend 30 minutes on V1. Get to V2 and V3.
• Invent and simplify. Your improvements should simplify, not complicate.
• Think big. V3 should show you can envision the system at massive scale.

Practice Template

For any system, practice this template:

Let me walk you through how this system evolves.

V1 (MVP): [Simple description]. This handles [X] and is sufficient for [Y] users.

V2 (Growth): When we hit [constraint], we add [improvement]. This brings us to [new capacity].

V3 (Scale): At [massive scale], we switch to [advanced approach]. This handles [Z] users with [latency/throughput].

Each phase adds complexity only when needed, keeping the system simple and maintainable.