Load Balancing and Autoscaling

Four instances sit behind your load balancer. Round-robin spreads requests evenly — one quarter each — and for a fleet of identical nodes serving uniform 40 ms calls, that is exactly right and costs almost nothing to compute. The trouble starts the day one endpoint does 3-second video transcodes and another does 12 ms JSON lookups: round-robin is request-blind, and even request counts say nothing about work.

Load balancing algorithms differ on one question: what does the balancer know when it picks a target?

Two entries deserve explanation because interviews ask about them by name.

Least connections sounds ideal and has a trap: connection count is a lagging, distorted proxy for load. During a deployment, a drained node holding three long-lived streams looks least busy and gets new traffic it should not receive. Production balancers therefore use weighted least-connections or power of two choices: sample two nodes at random and route to the less loaded one. P2C gets near-optimal balance in O(1) — no scanning all N nodes — and degrades gracefully instead of letting one bad node attract the whole herd. Envoy's LEAST_REQUEST and Finagle's default both work this way.

Consistent hashing answers a different problem: affinity with minimal disruption. Naïve hash(key) % N reassigns nearly every key when N changes (a deploy makes every session miss its cache). A consistent-hash ring with virtual nodes maps each key to its clockwise-successor node, so adding or removing one node of N relocates roughly 1/N of keys and nothing else — with 100 nodes, ~1% of sessions repin. Use it for session affinity, for sharded caches, and for rate-limit counters that must land on the same node; the virtual-node count (commonly 100–200 per physical node) is what keeps the ring balanced.

There is also a layer decision, separate from the algorithm: an L4 balancer (NLB, HAProxy TCP mode) forwards packets without looking at headers — millions of connections, ~sub-millisecond overhead, but it cannot route by path or terminate TLS. An L7 balancer (ALB, NGINX, Envoy) parses HTTP, so you get path-based routing, header canaries, cookie affinity, and per-route health — at the cost of a proxy hop (roughly 1–2 ms) and one more place for connection limits to bite.

Pick deliberately: uniform short API traffic starts with round-robin or P2C; sticky, connection-heavy traffic goes least-connections or P2C; anything that must recompute its own cache goes consistent hashing. And whatever you pick, it only matters if the balancer knows who is alive — which is the next chapter, and the mechanism that connects directly to the scaling decisions from Vertical vs Horizontal Scaling.

A new deploy goes out. The application boots in 40 seconds, its ORM connects to the database on the first request — which times out — and for two minutes every third user gets a 502. The pods were "healthy" the whole time: something was listening on port 8080. Port-open is not readiness, and conflating the two is the most common load-balancing failure you will be asked to diagnose.

Separate three questions that a production system asks at three layers:

Readiness is the important one. It should verify what your first request would need: database reachable within a budget (SELECT 1 under 200 ms), migrations applied, config loaded, cache warmed above a threshold, in-flight draining not started. It must stay cheap — a check that runs a heavy aggregation query adds load and, worse, creates correlated failure: every instance fails health because the database is slow, the balancer strips the fleet, and recovery requires traffic you no longer send. Check the dependency with a timeout; a health check that can hang is worse than no check.

Thresholds should be asymmetric on purpose — fail fast, recover slowly — so a single dropped packet does not flap a target out and back:

# typical ALB/NLB-style configuration
interval:              10s      # how often we ask
timeout:               5s       # how long an answer may take
unhealthy-threshold:   2        # 2 failures  ≈ 20s to pull a bad target
healthy-threshold:     5        # 5 successes ≈ 50s to admit it back
deregistration-delay:  30s      # keep serving in-flight work after SIGTERM

That asymmetry pairs with outlier detection in the mesh: Envoy ejects a host after, say, five 503s in a 10-second window and quarantines it for 30 seconds, so one sick node is handled locally instead of dragging every client into the failure.

Graceful shutdown is where fleets leak connections. On SIGTERM an instance should (1) fail readiness immediately so the balancer stops sending new requests, (2) keep accepting and finishing in-flight ones, (3) wait out the deregistration delay, (4) exit. In Go that is one line plus the signal handler:

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()
<-ctx.Done()          // readiness probe now returns 503; LB drains
_ = srv.Shutdown(ctx) // waits for in-flight requests, up to drain timeout

Skip this and every deploy closes sockets mid-request: clients retry, retries pile onto the remaining pods, and a routine rollout becomes an incident.

The two failure modes to name in an interview: too shallow (port check → user-visible 502s during boot and shutdown) and too coupled (health check hits a shared dependency without a timeout → the whole fleet is ejected together and nothing recovers). The fix is the same in both cases — readiness reflects ability to serve, with hard timeouts, asymmetric thresholds, and a drain window between "no new traffic" and "process exits."

Once health is trustworthy, the balancer's target list becomes the signal you can actually autoscale on — the subject of the final chapter here.

Your API nodes sit at 14% CPU while the p99 latency climbs from 180 ms to 2.4 s. The CPU-based autoscaler sees a relaxed fleet and does nothing; the real constraint is I/O wait and a saturated connection pool. This is the argument the key points keep making — scale on the metric that represents the bottleneck, not the metric that is easy to collect. CPU is correct for compute-bound work and misleading for anything waiting on a database, a downstream API, or a queue.

Three policy families cover most needs:

Target tracking is a thermostat: you declare "keep this metric at X," and the group computes desired capacity from the current ratio each evaluation period. The formula is worth memorizing:

desired = ceil(running × currentMetric / targetMetric)

# 10 instances at 80% CPU, target 40%
desired = ceil(10 × 80 / 40) = 20 instances

# ALB request-count-per-target: target 4,000 req/min per target,
# 6 targets currently serving 72,000 req/min → 12,000 per target
desired = ceil(6 × 12,000 / 4,000) = 18 instances   # 3× the pressure, 3× the targets

Target tracking handles scale-in symmetrically, which is why it is the default choice: it converges instead of ratcheting.

Step scaling reacts to alarm breaches with explicit adjustments: "above 500 concurrent connections → add 2 instances; above 1000 → add 4; below 200 → remove 1." It is precise and cheap to reason about, but it is open-loop — each step fires on a snapshot, and nothing guarantees the previous step has taken effect first.

Scheduled/predictive scaling pre-warms for known shapes: a flash sale at 09:00 gets capacity at 08:50, because instances need minutes to boot, pull images, and warm JIT and caches. Scale-out is not instantaneous — budget 2–4 minutes of launch plus 3–5 minutes of warmup before new capacity shows up in your metrics.

That lag is why cooldown arithmetic matters. Consider default-style settings: alarm period 60 s evaluated twice (2 min to breach) + 3 min to launch + 2 min for the new instances to register in the metric = ~7 minutes from decision to visible effect. With a 300-second cooldown, a second adjustment can fire before the first batch exists in the data, and you over-provision:

t=0    alarm breach, capacity 8 → +8 (target tracking, cooldown 300s starts)
t=300  cooldown over; metric still says "hot" because the 8 new nodes
       launched at t=120 are not warm until t=420 → +8 again → 24 nodes
t=500  metrics finally reflect the first batch; scale-in drags back down

The fix: set cooldown ≥ evaluation periods + launch time + warmup, or use a stabilization window (scale-in holds until the metric has been healthy across several consecutive periods). Also enable scale-in protection on instances you just launched for the first few minutes.

For asynchronous work, drop metrics entirely and scale on backlog: desired = ceil(queueDepth / maxBacklogPerInstance) — 120,000 pending jobs at 2,000 per instance means 60 instances, no guesswork. Prefer age of the oldest message over raw depth: a 200,000-message batch that drains in 90 seconds should not trigger a scale-out you pay for all hour.

The interview answer that lands: name the bottleneck metric first (requests, latency, queue depth, pool saturation), choose target tracking for anything with a natural per-unit target, size the cooldown from real boot+warmup timings you have measured, and treat CPU as one input among several. Next you decide where the served bytes should come from in the first place: Caching Strategies.