SLIs, SLOs, and Error Budgets

Measure the service users experience

A service-level indicator (SLI) is a measurable view of service behavior, ideally tied to a user journey. An availability SLI can be good requests divided by eligible requests. A latency SLI can be the share of valid requests completing below a threshold. Freshness, correctness, and durability may matter more than uptime for some products. Define the event population and exclusions carefully; a metric that omits difficult requests can report a healthy service while users fail.

A service-level objective (SLO) sets a target for an SLI over a time window, such as 99.9 percent of eligible requests succeeding over 30 days. A service-level agreement (SLA) is an external commitment with consequences; an internal SLO is an operational target and should usually provide margin before an SLA is breached. Do not set an SLO simply because a provider advertises a number. Choose a target that reflects user expectations, architecture capability, and the cost of additional reliability.

Calculate an error budget

The error budget is the allowed unreliability implied by the SLO. For a 99.9 percent availability objective over 30 days, 0.1 percent of the eligible period or events can fail under the chosen accounting model. In a simple time-based approximation, 0.001 × 30 days is about 43.2 minutes. Request-based SLOs use eligible event counts instead, so the allowed number of bad requests varies with traffic. State which model applies and how maintenance or invalid requests are handled.

The budget is a decision aid, not a quota to spend casually. Fast budget consumption signals risk to users and may justify pausing risky releases, investing in reliability, or reducing change rate. If the budget is healthy, teams can take measured product risk. Policies should be agreed across product and engineering so an exhausted budget leads to a known response rather than an argument during an incident.

Design a meaningful SLI

Choose good and total events at the user-visible boundary. A server process being up does not mean a checkout succeeded. Exclude only events outside the service's responsibility, and make exclusions auditable. For latency, define the threshold and denominator, account for successful responses only if that matches the user promise, and use percentiles or threshold ratios rather than averages that hide tails.

Use a sufficiently long window for objective reporting, often a rolling 28- or 30-day window, and shorter windows for alerting. Track SLIs by important user journey and service tier, not every internal endpoint equally. A small number of meaningful SLOs is more actionable than an exhaustive catalog with no decision policy.

Alert on burn rate

Burn rate compares the current error rate with the rate that would exhaust the budget over the objective window. Multi-window, multi-burn-rate alerts can detect both fast severe incidents and slower sustained degradation while reducing noise. A short window responds quickly; a longer window confirms that the impact is not a transient blip. The exact thresholds depend on service criticality and response capacity.

Alerts should page when human action is required. A dashboard can show budget remaining, while an alert points to a runbook and service owner. If the SLI pipeline is delayed or broken, its health must also be monitored; an empty data stream should not look like perfect reliability. Review objectives after product changes, traffic growth, or user feedback, but do not lower a target merely to silence alerts.

Make reliability policy collaborative

An error-budget policy might restrict high-risk deployments when the budget is exhausted, require a reliability review, or prioritize recovery work. Define exceptions for urgent security fixes and how they are recorded. Avoid a blanket release freeze with no path to restore service; reliability work itself may require deployment. SLOs coordinate decisions across teams when indicators, targets, and policy are transparent.

Define an SLO for checkout

Start with the user promise: a valid checkout request should complete successfully and quickly enough for the customer to finish an order. Define eligible requests at the API boundary and count success only when the order is durably accepted. A 200 response from a frontend that later loses the order should not count as good. Exclude malformed requests only when they are clearly outside the service contract and record the exclusion rule.

Choose an availability SLI such as successful eligible checkouts divided by total eligible checkouts, and a latency SLI such as the fraction completing under a threshold. Select a 30-day objective based on historical behavior, customer expectations, and any external commitment. If the objective is 99.9 percent, the request-based budget is one bad request per thousand eligible requests during the window. At one million monthly requests, that is about one thousand allowed bad events, not a fixed number of minutes.

Instrument budget and alerts

Emit numerator and denominator metrics from the same boundary and with consistent filters. Validate the SLI query against known incident periods and successful test traffic. Show the rolling objective, remaining budget, and burn rate on a service dashboard. Keep metric labels bounded and monitor the telemetry pipeline so missing measurements do not appear as zero failures.

Configure a fast burn alert for severe impact and a slower alert for sustained degradation. For example, a short window can page when budget is being consumed many times faster than planned, while a longer window confirms the trend. Tune thresholds against incident history and on-call response capacity. Link alerts to a runbook with dependency views and recovery steps. A warning that nobody can act on belongs on a dashboard, not in the paging channel.

Apply an error-budget policy

Agree on what happens when the budget is low or exhausted. The team may pause risky feature releases, require a review for changes that affect the SLI, or shift capacity to reliability work. Define how security fixes, data migrations, and recovery deployments are handled. Record exceptions and the decision owner. Do not block the deployment needed to fix the incident solely because the budget is exhausted.

Review the SLO with product and operations. If the objective is consistently missed, identify whether the target is unrealistic, instrumentation is wrong, or the architecture needs investment. If it is exceeded by a wide margin, the service may be over-engineered or the target may not reflect user expectations. Do not adjust the number without reviewing customer impact and service commitments.

Exercise the accounting

Replay a week of known request outcomes and verify the error budget calculation. Inject a short spike and a slower degradation to confirm each alert behaves as intended. Test a telemetry outage and ensure the SLO dashboard reports missing data rather than perfect availability. Review whether a retry or duplicate request counts once or multiple times according to the user promise.

Finally, ask whether the indicator can be gamed. If failed requests are excluded by a status filter, could the service return a different status while users still fail? If latency is measured only at one service, does it include queue delay? An SLI is trustworthy when its numerator, denominator, boundary, and data quality are explicit.

Avoid targets without user meaning

An SLO should represent a user-visible outcome, not merely a component's internal health. A server can return success while a downstream workflow is lost, and a dependency can be degraded without affecting the primary user path. Define the measurement boundary, eligible events, and what counts as good. Keep the number of objectives small enough that teams can act on them.

Targets that are too strict can drive unnecessary cost and slow product delivery; targets that are too loose fail to represent user expectations. Use service history, customer research, risk, and external commitments. An SLO is not a promise of perfection. It is a shared operating target with a budget that makes reliability trade-offs explicit.

Use budgets as policy, not punishment

An error budget communicates how much unreliability remains under the objective. When it burns quickly, teams may slow risky changes and focus on recovery. But a blanket freeze can prevent the deployment needed to fix the issue or address a security vulnerability. Define exceptions, decision owners, and a path for safe corrective changes. Review budget policy with product and engineering before an incident.

Burn-rate alerts balance fast response and noise. Very short windows catch severe spikes but can page on transient events; long windows are stable but may detect issues too late. Multi-window policies combine urgency and confirmation. Tune them using real incident patterns and the team's ability to respond. Monitor the SLI pipeline so missing data cannot masquerade as reliability.

Interview framing

In an interview, define an SLI at the user boundary, set an SLO and window, calculate the implied budget, and explain the response policy. Distinguish an internal SLO from an external SLA. Describe a burn-rate alert and how you would verify the numerator and denominator. Explain how product and engineering use a healthy or exhausted budget to choose between feature work and reliability work.

Also name a failure mode: excluding failed requests, measuring only average latency, or treating absent telemetry as success. This shows SLOs are measurement and decision systems, not a percentage chosen for a dashboard.

Calibrate objectives with customer evidence and service history. A target that simply matches the best month may be too strict during seasonal peaks; one based only on average behavior may tolerate unacceptable tail failures. Review the definition with product, operations, and support teams. Identify the user journey and the impact of a bad event before setting the threshold.

Budget consumption can be uneven. A deployment may spend a large share in a short incident, while slow degradation consumes it gradually. Multi-window burn-rate alerts help distinguish these patterns, but thresholds should be tuned to response time and false-page cost. Define who receives a warning and who receives a page. If the service has multiple SLOs, decide whether one exhausted budget blocks all changes or only changes that affect that journey.

An error budget is not a license to fail until it is empty. High-impact incidents may warrant immediate response even when the aggregate objective is healthy. Conversely, a fully consumed budget should trigger a risk discussion, not a ritual freeze that blocks reliability work. Record the exception rationale and follow-up actions. Track whether the policy changes release behavior and user outcomes; adjust the process if teams ignore it or apply it mechanically.

Review objectives when product behavior, traffic mix, architecture, or external commitments change. Preserve historical definitions so month-over-month comparisons remain interpretable. If an SLI query changes, recalculate the impact or mark a new measurement series rather than silently rewriting past performance. This keeps the objective accountable and prevents metric changes from hiding reliability trends.