Capacity Planning and Forecasting

Forecast demand, not just averages

Capacity planning estimates the resources needed to meet a workload while preserving reliability and acceptable cost. Forecasts should include growth, seasonality, launch events, geographic peaks, and workload mix. A monthly average hides an hourly spike; a daily average hides a batch window. Identify the time scale that drives saturation and collect telemetry at a resolution that can reveal it.

Model demand in workload units such as requests per second, concurrent sessions, queued jobs, storage bytes, or database operations. Separate average, peak, and burst duration. A system may handle a short peak with a queue but fail under a sustained peak. Include retry traffic and failover load because both can raise demand during an incident.

Translate demand into capacity

Use benchmarked cost per request, service time, or throughput per worker as a first estimate. Divide expected work by tested capacity and target utilization, then add headroom for variability and failures. This is a starting point, not a promise: performance changes with payload shape, cache hit rate, query mix, and contention. Validate with load tests representative of production.

For storage, forecast data growth and retention, then include indexes, replication, backups, and temporary migration space. For queues, compare arrival and processing rates and estimate drain time after a burst. For network, include cross-zone traffic and large response bodies. Capacity is a system property; scaling one tier can move the bottleneck to another.

Use scenarios and confidence

Maintain baseline, expected-growth, and stress scenarios. State assumptions and confidence for each. A launch forecast may be uncertain; use a range and identify the signal that will update it. Scenario planning reveals whether a design choice is robust or depends on one optimistic forecast. A forecast should be revisited when product demand, architecture, or provider limits change.

Headroom absorbs spikes and leaves room for failover, but excessive idle capacity increases cost. Set a target based on variability, recovery objectives, and scale-up delay. Autoscaling can reduce steady idle resources but requires metrics, limits, warm-up, and protection against oscillation. For capacity that cannot scale instantly, reserve enough headroom to cover the detection and provisioning interval.

Plan procurement and operational work

Track utilization, saturation, queue age, latency, and storage growth over time. Review forecasts at a cadence that matches procurement or architecture change lead time. Quarterly review may suit long-lead infrastructure; rapidly scaling cloud services need shorter feedback loops. Assign an owner to each forecast and document decisions such as purchasing commitments, expanding a partition, or changing retention.

Capacity is not only adding resources. Query optimization, caching, batching, backpressure, and workload scheduling can change demand or resource efficiency. Evaluate these options against complexity and reliability. A useful plan identifies the first limiting resource, the threshold for action, the time required to respond, and a fallback if growth exceeds the forecast.

Use more than one historical interval when seasonal patterns matter. Compare weekdays, weekends, launches, and known events, and note when product changes make old data less representative. A forecast should state its data window and confidence range.

Forecast a seasonal workload

Assume an event platform averages 1,000 requests per second, peaks at 4,000 during a weekly window, and has a major annual launch expected to reach 8,000. Workers process 250 representative requests per second at the chosen utilization target. A first estimate is 4 workers for average, 16 for the weekly peak, and 32 for the launch. Then adjust for startup time, failure headroom, workload variance, and the fact that one worker may be unavailable during deployment.

Do not use averages to size the launch path. Measure the peak duration and request mix. If the 8,000 rate lasts five minutes, a queue may absorb excess work if the product can tolerate delayed completion. If every request is interactive, more synchronous capacity or a graceful limit may be needed. Include retries and dependency failures in the stress scenario because they can amplify load.

Forecast storage and queue drain

Suppose the service stores 2 KB per event, receives 20 million events daily, and retains 90 days. Logical storage is roughly 40 GB per day, or 3.6 TB over 90 days before indexes, replication, and backups. Add headroom for migration and growth. Check whether old events can move to archival storage and ensure deletion is enforced throughout replicas.

For a queue, Little's law relates average items in the system to arrival rate times average time in system. If arrivals are 500 jobs per second and average processing delay is 20 seconds, average in-flight work is about 10,000 jobs. During a burst, estimate how fast the queue grows and how long workers need to drain it after arrivals return to normal. A sustained arrival rate above processing capacity cannot be solved by a finite buffer.

Track forecast error

Compare forecast and actual utilization, request rate, storage, and queue age. Record whether error came from adoption, seasonality, payload size, or an architecture change. Use forecast ranges rather than one number when customer behavior is uncertain. If actual demand reaches the upper scenario, define the trigger for capacity action and who owns it.

Review scale-up lead time. Cloud autoscaling may add instances in minutes, while database partitions, network quotas, or contractual capacity may require days. Maintain headroom for the slowest critical dependency. Test scale-out and failover under realistic load; a configured autoscaling policy is not evidence that the system can scale safely.

Make the plan actionable

Create a capacity table with metric, baseline, peak, forecast range, tested limit, target headroom, trigger, and owner. Link each assumption to telemetry or a benchmark. Include cost impact of the high scenario and the trade-off of reducing headroom. A quarterly review should produce an action or explicitly decide that no change is needed.

If growth exceeds the forecast, first verify telemetry and workload shape. Then identify the saturated component and evaluate scaling, optimizing, shedding optional work, or adjusting product behavior. A plan is useful when it provides time to act before users experience capacity failure, not merely when it predicts a number.

Measure scale-up time as part of the capacity budget. If a node takes several minutes to provision and warm, the system needs enough headroom to handle demand during that delay. Test quotas and dependency limits as well as compute, because a larger worker pool can expose a downstream bottleneck.

Headroom costs money and buys time

Capacity headroom absorbs spikes, failover, and provisioning delay, but idle resources have a cost. Set the margin from workload variability, scale-up time, recovery objectives, and failure-domain requirements. A stateless worker may scale quickly; a database or reserved network quota may not. Use different policies for different components instead of applying one blanket utilization target.

Autoscaling reduces steady idle capacity but can react too slowly, oscillate, or amplify a downstream failure. Define minimum and maximum capacity, warm-up, queue thresholds, and backpressure. Keep enough fixed capacity to handle the time before scaling takes effect. Exercise scale-down as well as scale-up so the system does not repeatedly discard warm caches or interrupt long-running work.

Forecast uncertainty explicitly

Seasonality, product launches, customer concentration, and workload mix can invalidate a straight-line growth projection. Use scenarios and sensitivity analysis. Identify which assumptions most affect architecture, and measure those more carefully. A forecast is a planning instrument that should be updated with actual demand, not a commitment that every peak will match the model.

Capacity and cost are linked. A high-headroom design may be justified by an availability objective, while a low-priority batch workload can use flexible or preemptible capacity. Include the cost and user impact of shedding optional work. An overloaded service that retries every request may spend more and perform worse than one that applies admission control.

Operationalize the plan

Assign owners, thresholds, and lead times. Track forecast accuracy and review resource limits before they become emergencies. Use load tests to validate key assumptions and production telemetry to update them. Include failure and recovery capacity, not only normal traffic. In an interview, show the demand model, peak and seasonal factors, headroom rationale, and a trigger for action. Explain what is measured versus assumed and how a miss changes the plan.

A forecast should describe both the expected demand curve and the consequence of being wrong. Define an upper scenario and the service behavior if demand exceeds it: autoscale, queue, degrade optional features, or reject safely. For each action, state lead time, cost, and the user-visible impact. This turns a forecast into a decision plan rather than a spreadsheet that is reviewed after capacity is exhausted.

Forecast error can be measured with absolute or percentage error, but percentage measures behave poorly near zero. Compare actual to forecast by horizon and workload class; an accurate monthly total may still miss the hourly peak. Preserve the forecast version used for purchasing or architecture decisions so later reviews can distinguish a bad model from a changed product assumption.

Capacity planning also needs governance. Define who approves headroom changes, who owns quota increases, and which dependency teams must be included. A capacity alert should fire early enough to cover procurement or provisioning lead time. If a resource is managed by another organization, include that lead time in the forecast and record the escalation route.

Review after traffic events and architecture changes, not only on a calendar. A new cache, index, or API consumer can change the resource relationship. Update the model when measured cost per request or processing time shifts. Track utilization and saturation together: low utilization with high latency may indicate a lock or dependency bottleneck rather than a need for more instances.