Vertical vs Horizontal Scaling

Your primary PostgreSQL node sits at 82% CPU through the evening peak. You resize from db.m6g.2xlarge (8 vCPU, 32 GiB) to db.m6g.8xlarge (32 vCPU, 256 GiB), latency recovers, and the team stops thinking about capacity. Three months later the same graph points upward, and this time the largest instance your region sells is within spitting distance of what you think you need. Vertical scaling — making the single node bigger — ran out of road.

Vertical scaling is the default first move for stateful systems for a good reason: it needs no code changes, no data redistribution, no coordination between replicas. One parameter flips, one failover later, you have 4× the CPU. But every rung on the instance ladder costs more per unit of capacity than the one below it, and the top of the ladder is a hard wall.

Instance          vCPU   RAM (GiB)   On-demand $/hr (us-east-1, approx)
m6i.large            2          4          0.096
m6i.xlarge           4          8          0.192
m6i.4xlarge         16         64          0.768
m6i.16xlarge        64        256          6.144
m6i.metal          128        512         12.288

The dollar cost is the least interesting part. Three other costs bite harder:

• You hit a hardware ceiling you cannot buy past. Largest general-purpose offerings top out around 448–512 vCPU and 3–8 TiB of RAM. If your working set or request rate outgrows that, there is no rung eight.
• Software scales sublinearly with the box. A JVM with a 64 GiB heap pays multi-second GC pauses you never saw at 8 GiB. PostgreSQL's shared buffers, single-writer WAL, and max connection limits do not care that you doubled the cores. Lock contention inside a process gets worse with more cores chasing the same mutex — the classic scaling wall shows up as CPU parked in kernel futex waits while the box looks idle.
• The node becomes a single point of failure with a maintenance window. Resizing a database means a failover or a restart; a restart is an outage for anyone with open transactions. And with one node, there is no partial degradation — a lost box is a lost service. Capacity is also bought for peak and paid for 24/7: a diurnal workload at 30% average utilization still keeps the 3 AM-sized machine running.

Amdahl's law puts a number on the first software limit. If 5% of your request path is serialized (a global lock, a single-threaded serializer), the theoretical maximum speedup from infinite cores is 1 / 0.05 = 20× and long before that, contention dominates. Doubling cores rarely doubles throughput on a lock-heavy service. The cheap pre-purchase test is empirical: run your load test at 2× the cores and check whether throughput moved. If it improved by 15% instead of 90%, you have found your serialized fraction, and the next three sizes will disappoint you the same way.

So vertical scaling is not wrong — it is the right tool until the ceiling, and the wrong tool as a strategy. It wins for single-writer datastores (a relational primary is a single writer no matter how many machines you own), for software licensed per core where doubling cores doubles the license bill, and for small fleets where operating four boxes costs more engineer time than one box costs in dollars.

The interview version of this question is rarely "which is better." It is: where does scale-up stop working, and what do you do next? The answer is the subject of the next chapter — Load Balancing and Autoscaling is only possible once the tier you are scaling is built from interchangeable, stateless nodes. The state question comes first.

You double your web pods from 4 to 8 behind the load balancer. Half of your users are now logged out at random, checkout carts empty on refresh, and file uploads fail intermittently. Nothing is broken in the traditional sense — every pod is healthy. The state simply lives in the wrong place: inside one process, reachable only by the requests that happened to hit it.

Horizontal scaling adds identical nodes instead of enlarging one. It only works under a strict precondition: any node must be able to serve any request. That single sentence is what "stateless tier" means, and it is the prerequisite for cheap scale-out called out in every architecture review.

Inventory what typically hides in process memory, then move it:

Sticky sessions look like a fix and are usually a trap: they pin a user to a pod so in-memory sessions keep working, but now scaling out redistributes only new users, a dead pod drops its whole shard of sessions, and any rebalancing requires session migration. Keep affinity only where physics demands it (long-lived WebSocket connections), and use consistent hashing so that removing one node relocates about 1/N of connections instead of all of them — you will see that algorithm properly in Load Balancing and Autoscaling.

The subtler cost of scale-out is that every node carries the full fixed overhead, and shared backends feel the sum of all pools. PostgreSQL's default max_connections is 100. One node with a 50-connection pool was comfortable; ten pods with the same pool want 500 connections and will exhaust the primary unless you introduce a pooler (PgBouncer in transaction mode) or shrink per-pod pools. Budget the arithmetic before you autoscale:

per-node pool 20 connections × 20 pods  = 400 backend connections
PostgreSQL default max_connections      = 100   ← you are already broken at 5 pods
fix: PgBouncer (transaction pooling) or pool = floor(100 / (pods × safety))

The payoff is real, though. Capacity becomes additive: four nodes at 500 requests/second gives ~2000 req/s, and failure becomes partial — lose one node of ten and the load balancer routes around it while you lose 10% of capacity, instead of 100%. That is the difference between a scaling model and a disaster-recovery story.

Two practical consequences follow. First, boot is now a capacity event: a fresh node starts with cold caches and empty JIT profiles, so expect several minutes of degraded latency after scale-out — warm pools and gradual traffic ramps exist for exactly this reason. Second, memory is no longer shared for free: ten JVMs with 2 GiB heaps cost 20 GiB where one 8 GiB heap did it, because each node caches its own copy of hot data and holds its own runtime overhead.

If you take one thing from this chapter: horizontal scaling is not "add machines," it is make the node disposable. Anything that survives only inside one process is a blocker to scale-out and should be treated as a design defect, not an implementation detail.

"Scale up or scale out?" is a standard senior-level interview question, and the weak answer picks one and defends it. The strong answer starts from the workload: what kind of state does it hold, how does it fail, and how does demand curve over time. After API Gateways and Backend-for-Frontend taught you to peel cross-cutting concerns off the app tier, this is the decision you apply to what is left.

The economics deserve concrete numbers. Two capacity plans can cost the same and behave nothing alike:

Plan A (vertical): 1 � m6i.4xlarge  16 vCPU, 64 GiB   $0.768/hr
Plan B (horizontal): 4 � m6i.xlarge  4 vCPU, 16 GiB   4 � $0.192 = $0.768/hr

Same price. Plan A: one failure domain, capacity fixed at 16 vCPU, resize = restart.
Plan B: 4 failure domains, scale 4?8 for the peak and back, deploy = rolling, lose a node = lose 25%.

For a workload with a 5� peak-to-trough ratio, Plan B also wins on utilization: at trough you can run two nodes and scale to ten during the peak, while Plan A must be sized for the peak permanently � a 16-vCPU box idling at 20% for eighteen hours a day. That elasticity argument is the reason almost all application tiers go horizontal on sight, and it is what the autoscaling policies in Load Balancing and Autoscaling exploit.

The honest counterweight is everything scale-out makes you pay. Each node duplicates its runtime overhead: a 1 GiB heap plus a sidecar at 100 MB per pod means 5 replicas burn ~5.5 GiB doing what one node did with 1.2. Serialization replaces function calls � remote state lookups cost 0.5�2 ms where an in-process map costs microseconds. Cache hit ratios can fall when a shared cache's keyspace is partitioned across nodes and hot keys overload their shard. Distributed correctness (exactly-once delivery, cross-node locks) is now your problem. If a system is small, single-digit RPS, and boring, one well-chosen box and a backup plan is genuinely the better engineering answer.

A decision procedure you can state in an interview:

1. Classify the state. None (or externalizable) ? horizontal, full stop. Single-writer, strongly consistent ? vertical first.
2. Plot the demand curve. Spiky or diurnal ? horizontal, because you pay for elasticity. Flat and modest ? vertical may be cheaper in total cost of ownership.
3. Find the ceiling. Largest instance you can buy � current size = your runway in months. When runway drops under a quarter, start the sharding or replica design.
4. Price the failure. What does an unplanned restart cost in money and SLO? If the answer is "an outage," that is a strike against a single node regardless of size.

The synthesis to carry into the room: scale vertically where the data model forces a single writer, scale horizontally everywhere else, and treat statelessness as the enabling condition � not a stylistic preference � for everything that follows in this phase.