Bulkheads and Rate Limiting

Your service has a single worker pool of 200 threads serving every route. A new /reports endpoint runs 5-second analytical queries. One batch job fires 300 report requests, all 200 threads are borrowed by reports, and /checkout — which needs 20 milliseconds of local work and a healthy database — queues behind them and times out for every customer. Nothing is broken: the database is fine, checkout's own code is fine, and utilization graphs look busy rather than wrong. The service sank because one workload was allowed to consume all of the shared capacity.

A bulkhead is the ship-building idea applied to threads: partition the pool so that saturation in one compartment floods only that compartment. In practice you split worker threads (or connection-pool slots, or semaphores) per route, per dependency, or per tenant, and cap each partition independently:

Bulkhead reports = Bulkhead.of("reports",
    BulkheadConfig.custom()
        .maxConcurrentCalls(40)                       // reports may never exceed 40 in flight
        .maxWaitDuration(Duration.ofMillis(50)));      // beyond that, reject fast

Bulkhead checkout = Bulkhead.of("checkout",
    BulkheadConfig.custom()
        .maxConcurrentCalls(80)
        .maxWaitDuration(Duration.ofMillis(100)));

// a call outside its partition's allowance fails immediately with BulkheadFullException

The arithmetic is why this works rather than being a preference. With one shared pool of 200 threads and a 5-second route, that route's throughput ceiling is 200 / 5 s = 40 req/s — and while it sits at that ceiling, every other route gets zero threads. With reports capped at 40 and checkout holding 80, reports still tops out at 40 / 5 s = 8 req/s while checkout keeps 80 threads free for its own 20 ms calls: capacity 80 / 0.02 s = 4,000 req/s. Same machine, same total threads, one partition — and the isolation survives any amount of report traffic.

Bulkheads show up everywhere once you know the name. The service mesh caps concurrent streams per upstream; a database access layer partitions connections between read and write pools so a read storm cannot block commits; Kubernetes gives each tenant or workload its own pod so noisy-neighbor eviction is a scheduling decision rather than a thread fight; gRPC enforces max_concurrent_streams per HTTP/2 connection; a job queue runs bulkheads of workers per queue so nightly ETL cannot delay payment processing. In every case the mechanism is the same: a hard per-class limit on concurrency, enforced before the work starts.

The design decision is what happens when a partition fills, and there are only three honest answers. Reject immediately (429 or 503) — predictable latency, explicit signal, what most public APIs should do. Queue with a bounded wait — acceptable when the queue is small and its depth is monitored, because a queue is just latency you have not charged for yet. Or spill to a degraded mode — which is exactly the next topic. The answer that must be rejected outright is an unbounded queue: it converts a capacity problem into a memory problem and hides the overload until the OOM killer arrives.

Bulkheads are not free. You pay in utilization — a partition idling while another saturates is "wasted" headroom, and that is the price of isolation, not a bug to optimize away. You pay in configuration: someone must decide the numbers, and the defaults should come from measured capacity (pool × service_time = expected throughput), not guesses. And you must guard against over-partitioning, where twenty tiny pools reject while the host sits at 40 % CPU.

In an interview, the question is usually "how do you stop one slow dependency from taking down the whole service?" The complete answer names two mechanisms and their order: bulkheads cap how much of your capacity each dependency may occupy; rate limits cap how much of your capacity any caller may demand. The second half lives in the rest of this topic, after you have seen how the same breaker-oriented view from Circuit Breaker Pattern limits calls outward rather than capacity inward.

Bulkheads protect you from your dependencies and your own routes. Rate limiting protects shared capacity from callers — one tenant's retry-happy script, one partner's batch job, one DDoS — by deciding how many requests may pass in a given period. The choice of algorithm is a choice about what contract you are actually promising, because the algorithms differ in exactly one dimension: how they treat bursts.

The token bucket is the one you should be able to derive from memory. Each key holds a bucket of capacity B that refills continuously at r tokens per second; a request costs one (or more) tokens; a request with an empty bucket is rejected with 429. State is two numbers — current tokens and last refill timestamp:

def allow(bucket, capacity, refill_per_sec, cost, now):
    elapsed = now - bucket.ts
    bucket.tokens = min(capacity, bucket.tokens + elapsed * refill_per_sec)
    bucket.ts = now
    if bucket.tokens >= cost:
        bucket.tokens -= cost
        return True
    return False

This yields the contract most product teams actually want: "20 requests per second sustained, with a burst of up to 100 to absorb page loads and retries." A fixed window cannot express that — it would either allow 100 continuously or forbid bursts entirely. Its known defect is the boundary double-spend: a client may fire its full quota at 09:59:59 and again at 10:00:00, seeing 2× the intended rate over any two-second span. The sliding window counter fixes most of that by weighting the previous window's count by its overlap with the current one (in Redis, count = prev × (1 − elapsed/window) + current), keeping memory at O(1) while removing the cliff. The sliding window log is exact but stores a timestamp per request, which is fine at 5 requests/second and ruinous at 50,000.

A Redis token bucket must be atomic or it is not a limiter: two app instances reading the same key must not both spend the last token, so the refill-and-spend runs as one Lua script executed by a single Redis node:

-- KEYS[1]=bucket, ARGV = capacity, refill/sec, cost, now_ms, key_ttl
local t = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(t[1]) or tonumber(ARGV[1])
local ts     = tonumber(t[2]) or tonumber(ARGV[4])
tokens = math.min(tonumber(ARGV[1]), tokens + (ARGV[4] - ts) / 1000 * tonumber(ARGV[2]))
if tokens < tonumber(ARGV[3]) then return 0 end
redis.call('HMSET', KEYS[1], 'tokens', tokens - ARGV[3], 'ts', ARGV[4])
redis.call('PEXPIRE', KEYS[1], ARGV[5])
return 1

Choose by contract, not by habit: user-facing API with bursty clients → token bucket; daily or monthly quota, billing, or abuse detection → window counter; feeding a slow downstream where smoothness matters more than allowance → leaky bucket (it is a rate shaper, output at constant rate, rejecting or queueing whatever cannot fit). If you need fairness across many clients rather than a global cap, add a per-key bucket inside a weighted-fair queue.

One last interview detail: randomized refill or jittered admission is often worth adding on top, for the same reason backoff jitter helps — synchronized clients hitting a synchronized window boundary re-create the thundering herd you were limiting in the first place.

A single tenant drives 40 % of your traffic with a polling loop. Global rate limiting does nothing for you, because the tenant is within the global limit while everyone else's latency craters. The fix is hierarchical enforcement keyed by identity: a fleet-wide ceiling, per-tenant or per-API-key budgets underneath it, and per-endpoint budgets under those. The strictest matching limit wins, and enforcement must be keyed by who is asking — a shared pool with a shared counter protects capacity but provides no fairness.

Not all requests cost the same, so budgets should be weighted. A GET /health and a POST /search that fans out to six shards must not draw from the same one-token allowance:

limits:
  - match: { path: "/health" }        cost: 1
  - match: { path: "/v1/search" }     cost: 8    # fan-out, unindexed terms cost 16
  - match: { path: "/v1/orders" }     cost: 2
tenant:
  burst: 100        # tokens
  sustained: 20     # tokens/second

Cost-weighting makes the limit an honest reflection of resource consumption instead of a request counter that can be gamed by choosing cheap-looking endpoints.

The client contract is half the feature. Return the standard headers on every response — RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset (the IETF draft conventions) — and on rejection return 429 Too Many Requests with Retry-After. That 429 is precisely the status your retry policy from Timeouts and Retries is allowed to honor; a client that ignores it will retry into the same wall and burn its retry budget. Document the burst size, because clients that know they may burst 100 will back off smarter than clients guessing at your ceiling.

Getting 429 and 503 straight is a reliable interview question:

Where you enforce depends on cost. The gateway enforces cheap, uniform, per-key limits — it has the API key and no application context. The service enforces expensive, cost-weighted limits — only it knows that this search fans out. Enforcement placement is a latency-versus-accuracy trade: a centralized Redis limiter is exact but adds a round trip and makes availability of your limit depend on Redis; local in-process counters with periodic cross-node sync add no latency but only approximate the limit, allowing brief overshoot as fleet nodes drift. For abuse protection, approximation is fine — accept a few percent of overshoot. For contractual billing quotas, converge on the centralized (or synchronized) option.

Publish the limits. A limiter without published per-endpoint numbers and a documented burst size forces every integration partner to discover your ceiling by trial and error, and their discovery process looks exactly like abuse in your dashboards. Put the contract in the API description (x-ratelimit-limit, x-ratelimit-burst) and treat a limit change as a breaking change with the same deprecation care you give a field removal.

Fairness at the queue level is the last layer: when a partition is full, a single FIFO lets one tenant's flood occupy every slot. Weighted-fair or round-robin queues over per-tenant sub-queues ensure each tenant gets its share of the bulkhead rather than first-come-first-served victory.

By now you have capped dependencies (bulkhead), capped callers (rate limit), and bounded each route's concurrency. What none of these do is decide which work to abandon when the whole system is past saving — that is the job of the next topic, Graceful Degradation and Backpressure.