Sizing Compute, Queues, and Storage

Size from a workload model

Infrastructure sizing starts with workload and service objectives: request rate, payload size, concurrency, processing time, retention, durability, and peak shape. Use measured cost per request or throughput per worker where possible. State assumptions and calculate an order of magnitude before selecting resource counts. A benchmark on a different payload or cache state may not predict production capacity.

For compute, a first approximation is requests per second times CPU seconds per request. If a handler uses 20 milliseconds of CPU at 500 requests per second, it consumes about 10 core-seconds per second, or 10 fully utilized cores. A target utilization of 60 percent implies more provisioned cores, before accounting for I/O wait, uneven load, deployment headroom, and failure. Use this calculation to anchor a load test, not as a final instance count.

Concurrency follows arrival rate and service time. Little's law states average work in a stable system equals arrival rate multiplied by average time in the system. At 200 requests per second and 0.5 seconds average service time, about 100 requests are in flight on average. Tail latency and bursts can create much higher concurrency, so size connection pools and worker queues using distributions and tested peaks.

Size queues for bursts and recovery

A queue absorbs temporary mismatch between arrival and processing rates. If arrivals are 600 jobs per second and workers process 500, backlog grows by 100 per second. A finite buffer can cover a burst but cannot solve sustained overload. Calculate backlog growth and drain time after the burst, including the requirement that processing capacity exceed continuing arrivals. Monitor oldest-message age, not only depth, because age maps more directly to delayed work.

Retries can multiply demand during an outage. Use bounded attempts, exponential backoff with jitter, idempotency, and a dead-letter strategy. Estimate the queue with retry volume and poison messages. A queue limit may protect storage but must have a defined behavior when reached: reject, shed low-priority work, or apply backpressure to producers.

Size storage beyond logical data

Logical storage is records per unit time multiplied by average record size and retention. Physical capacity also includes indexes, replication, write-ahead logs, snapshots, backups, temporary migrations, and free-space requirements. State units and retention explicitly. Include growth and deletion lag. If data moves to archive tiers, include retrieval and query costs as well as storage.

For a database, index size and write amplification can be significant. Measure representative records and query plans. Replication improves durability or read capacity but multiplies storage and network. Backups need restore capacity and recovery time, not just a storage budget. Test restoration at the scale the architecture expects.

Account for bottlenecks and failure domains

Scaling one tier can shift pressure to another. More API workers may exhaust database connections; more consumers may overwhelm a downstream provider. Size end-to-end and define concurrency limits, bulkheads, and backpressure. Reserve headroom for rolling deployments and a failed availability zone. Ensure capacity assumptions hold after a node or region is lost if the recovery objective requires it.

Sizing is iterative. Estimate, benchmark, load test, observe, and revise. Record the tested workload, software version, data shape, and saturation point. A number without its workload context cannot be reused safely. The purpose is to identify likely limits early and leave enough time to change architecture before users encounter them.

Estimate a worker service

Assume a worker processes 300 jobs per second at 25 milliseconds of CPU per job. Ideal CPU demand is 7.5 core-seconds per second. At 50 percent target utilization, provision an initial estimate near 15 cores, then account for I/O, garbage collection, uneven job cost, deployment overlap, and a lost worker. Benchmark the real job mix because a heavy-tailed payload changes the average substantially.

If average service time is 0.4 seconds and arrival rate is 300 jobs per second, Little's law suggests about 120 jobs in the system at steady state. This does not size the queue for a burst. If arrivals rise to 450 while processing remains 300 for ten minutes, backlog grows by 90,000 jobs. After the burst, workers must process faster than arrivals to drain it. Calculate drain time and the user-visible delay before calling the queue large enough.

Include retries and availability

If a downstream service times out, retries can increase arrival rate. Estimate normal attempts and worst-case attempts under the retry policy. Use exponential backoff and jitter, cap retries, and send persistent failures to a dead-letter queue with an owner. Make processing idempotent so redelivery does not duplicate side effects. Test whether the queue and downstream service can recover without creating a retry storm.

Plan for one worker host or availability zone to fail. If normal capacity is spread evenly across three zones, determine whether the remaining two can handle the required workload. A design sized exactly to average demand may fail during maintenance or rolling deployment. Include a minimum safety margin and validate the cost of that headroom against the recovery objective.

Calculate retained storage

Suppose 5 million records arrive daily at 2 KB each and are retained for 180 days. Logical storage is about 10 GB daily, or 1.8 TB across the retention window. Add indexes, replication, backups, logs, and temporary migration space. Measure actual serialized size because compression and metadata can change the result. Include a deletion lag if expired data remains in backups or replicas for a period.

For a relational database, estimate index growth and write amplification using a representative schema and query mix. For object storage, estimate list and retrieval operations as well as bytes. If data is archived, calculate restore time and egress before claiming it is available for an incident. Test backups by restoring a representative volume, not only by confirming that a snapshot exists.

Validate with a bottleneck test

Load test the whole path and observe CPU, memory, connection pools, queue age, database latency, and downstream throttles. Increase one load dimension at a time where possible, then test the realistic combined peak. Identify the first saturation point and how performance degrades. A service that meets throughput only when every dependency is healthy may not meet its availability target.

Record the workload generator, payload distribution, test duration, software version, and warm-up. Compare observed throughput with the estimate and update the model. If the bottleneck changes after scaling one resource, revise the system estimate rather than scaling blindly. Capacity sizing is complete when limits, headroom, failure behavior, and cost are understood for the target workload.

Capacity math is a starting point

Requests per second multiplied by CPU time per request gives a rough compute requirement, while Little's law estimates steady-state work in progress. Neither accounts for every tail, burst, dependency, or failure. State units, utilization target, data shape, and assumptions. Validate with a representative load test and use actual production telemetry to revise estimates.

Queue depth is not the same as queue health. Oldest-message age and drain time describe the user impact. A buffer protects against a temporary burst but increases latency and storage. Sustained arrivals above processing capacity eventually fill any finite queue. Define backpressure, rejection, priority, and retry behavior before relying on buffering.

Storage calculations must include replication, indexes, backups, retention, and restore. Replicas and backups have different purposes and failure modes. A retention policy should include deletion from secondary copies according to the required lifecycle. Test restore time at realistic scale because capacity that cannot be recovered within the objective is not useful resilience.

Size for failure and rollout

Include headroom for a lost node or zone, rolling deployment, and scale-up delay. The remaining capacity must meet the service objective after failure, not just the average when everything is healthy. Autoscaling can add resources but needs warm-up and quotas. Rate limits and backpressure protect dependencies from overload.

Observe the system as a whole. Compute can be underutilized while a database or provider is saturated. More consumers may worsen downstream throttling. Use bottleneck analysis to identify the first constraint and understand how scaling moves it. In an interview, show the estimate, failure case, queue math, storage factors, and validation plan. This demonstrates sizing as an iterative system decision rather than arithmetic in isolation.

Sizing assumptions should include operational limits such as database connection ceilings, provider quotas, file descriptors, and network bandwidth. A compute estimate that fits CPU can still fail when all workers open a connection per request. Use connection pooling, concurrency limits, and backpressure, then test them under the expected mix. Resource quotas may require a lead-time request before a launch.

Document both steady-state and degraded-state sizing. If one zone is lost, the remaining workers may need to process full traffic while a database replica is promoted. A recovery plan that restores service but leaves no capacity headroom may fail again during the next spike. Include deployment overlap and background compaction or backup traffic in the calculation.

Load tests should capture saturation behavior, not just maximum throughput. Note the point where p95 latency rises, queues begin growing, or error rates increase. Use those thresholds to set scaling and admission-control policies below the hard limit. Re-test after provider, schema, or query changes that alter resource cost per operation.

In a design review, ask an operator to explain how each sizing assumption will be monitored. Identify the alert, its threshold, the owner, and the action it triggers. If no one can name the response, the capacity number has no operational meaning. Preserve the test conditions and evidence so a future team can tell whether the benchmark still applies.