Observability Stack Architecture

Design the telemetry path end to end

An observability stack collects signals from applications and infrastructure, processes them, stores them, and makes them queryable for operators. OpenTelemetry provides APIs, SDKs, semantic conventions, and a collector pipeline so teams can instrument once and export to compatible backends. It reduces instrumentation lock-in, but it does not make every backend query, dashboard, retention policy, or cost model interchangeable.

A common path is application SDK to a local or regional collector, then through processors to one or more exporters. Receivers accept OTLP or other formats. Processors can batch, enrich, redact, sample, or limit attributes. Exporters send metrics, logs, and traces to their storage systems. Use the collector as a managed boundary for credentials and routing rather than embedding backend secrets in every service. Keep application instrumentation focused on domain and service boundaries; central processing should not be used to invent semantics that only the service understands.

Select collection topology

An agent or sidecar collector can provide local buffering and metadata enrichment, but it adds deployment resources and upgrade work. A node-level collector reduces per-process overhead and can centralize host signals, while a gateway collector can apply shared policy and fan out to backends. A gateway is simpler to manage centrally but becomes a network dependency and potential bottleneck. Many systems use a layered topology with local collection and regional gateways.

Define backpressure behavior. Telemetry export should not block business requests indefinitely when a backend is unavailable. Configure bounded queues, retry limits, memory limits, timeouts, and drop policies. Monitor dropped spans, rejected metrics, queue utilization, and export errors. A collector that silently drops data can make an incident harder to diagnose precisely when volume spikes.

Match storage to signal shape

Metrics are time series optimized for numeric aggregation and alerting. Their label cardinality drives storage and query cost, so use bounded dimensions. Logs are event records that benefit from indexing and retention controls; index fields that support common investigations rather than every arbitrary attribute. Traces are linked spans with sampling and trace-aware lookup. A single database rarely optimizes all three workloads equally.

Connect backends through common identifiers and consistent resource attributes: service name, environment, region, and version. Use trace ids in logs and exemplars from metric histograms when supported. Avoid putting high-cardinality values into metric labels simply to create links. Separate raw detailed data from aggregate views, and define which teams can query each level.

Govern access, privacy, and retention

Telemetry can include sensitive payloads, identity claims, and infrastructure details. Redact or drop secrets before export, allowlist attributes, limit untrusted baggage, and restrict access by team and environment. Configure encryption in transit and at rest. Set retention according to investigation needs, legal obligations, and storage cost. Audit access to raw events and ensure deletion propagates to replicas and exports where required.

Tenant isolation matters in shared platforms. Use authenticated ingestion, tenant-aware routing, and query permissions that prevent one team from reading another team's data. Avoid treating a service name or label as authorization. Keep collector credentials scoped to the destination and protect configuration that can redirect telemetry to an unauthorized endpoint.

Operate the observability platform

Dashboards should answer operational questions: service golden signals, dependency health, queue age, and reliability objectives. Provide a first screen that helps an on-call responder orient, then link to detailed logs and traces. Maintain a service catalog and ownership map so alerts route correctly. Version dashboards, alerts, and collector configuration alongside service changes where practical.

The stack itself needs service indicators. Track ingest volume, accepted and dropped telemetry, query latency, storage growth, and exporter health. Estimate cost by signal type, cardinality, sampling rate, retention, and query load. OpenTelemetry standardization helps portability at the instrumentation layer, while storage and operational choices remain deliberate architecture decisions.

Build a collector pipeline

For a set of services running in two regions, deploy an OpenTelemetry collector close to workloads and a regional gateway. Applications export OTLP to the local collector. The first processor batches data and adds resource attributes such as service name, deployment environment, region, and version. A privacy processor removes disallowed fields and limits attributes. The gateway routes metrics, logs, and traces to their appropriate storage backends.

Keep the topology proportional. A small deployment may use one collector tier; a large multi-tenant platform may need regional gateways for network control and isolation. Avoid putting a central gateway on the synchronous request path. The SDK should export asynchronously with bounded queues and timeouts so backend outages do not stall application traffic.

Set resource and cardinality policy

Define required resource attributes and validate them at ingestion. A service missing its environment or version should be detectable. Allow metric labels such as route template, status class, and region, but reject request ids, user ids, raw URLs, and arbitrary exception text as dimensions. Route high-cardinality details to sampled traces or structured logs with access controls.

Estimate ingest cost from event volume, average record size, sampling ratio, cardinality, retention, and query demand. A trace-heavy service with large payload attributes can dominate cost even when its request rate is moderate. Use a tail-sampling policy for errors and slow traces only if collectors can buffer complete traces and route them consistently. Track dropped and sampled-out data so the team knows the limits of visibility.

Provide safe access and retention

Require authenticated ingestion and scoped credentials from collectors. Keep destination credentials in a secret store and restrict collector configuration changes. Separate raw event access from aggregate dashboards. Apply retention by signal and environment: production traces may need a shorter detailed window than compact SLO metrics. Redact sensitive attributes before export, and periodically test that access policies prevent cross-team queries.

Create a service dashboard with request rate, errors, latency, saturation, and dependencies. Link a latency panel to representative traces and a trace id to structured logs. Provide runbook links and service ownership. Do not create a dashboard per team with incompatible definitions of the same metric; maintain shared semantic conventions and a review process for breaking changes.

Test failure and cost controls

Disable a backend exporter in a staging environment. Confirm queues remain bounded, application traffic continues, drops are counted, and recovery drains within an acceptable time. Generate high-cardinality labels and verify the pipeline rejects or transforms them. Inject a secret-like test value and ensure redaction occurs before it reaches persistent storage. Exercise a trace sampling rule with successful, slow, and failed operations.

Review collector CPU, memory, queue length, export latency, and drop rate under a traffic burst. Set limits and alerts before rollout. A central observability stack is itself production infrastructure; plan upgrades, capacity, failover, and access recovery. The test is successful when teams retain enough useful telemetry to diagnose issues without allowing collector failure or unbounded data to threaten the services being observed.

Standardize without pretending backends are identical

OpenTelemetry can reduce instrumentation lock-in through common APIs and protocols, but backend query languages, storage formats, alerting, and retention differ. Preserve a portable instrumentation layer while making backend-specific capabilities explicit. Evaluate the cost of migration and data export before assuming a new exporter makes a full backend replacement easy.

Collector placement balances local resilience, central governance, network cost, and operational burden. Agents provide local buffering and context; gateways centralize routing and policy but can become a bottleneck or dependency. Define resource limits, queue behavior, retry policy, and what happens when an exporter is unavailable. Monitor the collectors themselves so telemetry loss is visible.

Control cardinality, privacy, and spend

High cardinality can make a metrics backend expensive or unusable. Keep identifiers bounded and route request detail to logs or sampled traces. Sampling lowers storage cost but can hide rare failures; tail sampling preserves interesting traces at the cost of buffering and routing complexity. Set retention by operational value and compliance, and remove sensitive data before export rather than relying only on access controls.

A shared stack creates a multi-tenant security boundary. Authenticate ingestion, scope credentials, isolate query permissions, and audit access. Ensure collector configuration cannot be changed by untrusted service code to exfiltrate telemetry. Cost allocation by team or service helps owners understand the effect of cardinality, sampling, and verbose logging.

Interview framing

In an interview, draw the path from SDK through collectors and processors to signal-specific storage. Explain agent versus gateway placement, backpressure and drop behavior, and how trace ids connect signals. Discuss data privacy, tenant isolation, cardinality, retention, and cost. Finish with stack health indicators and ownership. This presents observability as a governed data platform with explicit failure behavior, not a collection of dashboards.

Backend migration deserves a plan. Dual export can compare dashboards and query behavior, but it temporarily doubles ingest cost and may expose data to a second vendor. Define which data is copied, how long both systems run, and what result permits retirement of the old backend. Export formats and query semantics may differ, so test representative alerts and historical comparisons before declaring parity.

Failure isolation is important. If one exporter is unavailable, avoid blocking all signals behind a single queue. Use bounded per-destination queues or routing so a slow log store does not prevent SLO metrics from arriving. Set retry limits and report destination-specific drops. Decide whether critical security events require a durable path separate from sampled operational telemetry.

Changes to semantic conventions should be versioned. A service rename or attribute change can split dashboards and break alerts. Provide migration aliases or update queries atomically. Assign owners to shared collectors, dashboards, and policies, and maintain a catalog that links signals to a service team. Review access whenever team ownership changes.

During an incident, operators need an alternate path if the observability backend is the failing system. Keep infrastructure health metrics and collector logs available independently where possible. Document how to inspect local buffers and how to restore access. A centralized stack is a dependency, so its own recovery objectives and failure modes belong in the architecture review.