Logs, Metrics, and Traces

Choose the signal that answers the question

Observability signals help an operator infer what a system is doing from its emitted data. Metrics, logs, and traces answer related but different questions. Metrics aggregate measurements over time and are efficient for alerting, trends, and service-level indicators. Logs record discrete events with fields that explain what happened for a particular operation. Traces connect timed spans across components to show where one request spent time and where it failed.

For a sudden increase in failed requests, a metric can show when the rate changed and whether it affects one region. A log can reveal a specific error code and request context. A trace can show that the API waited on a database or downstream service. Using only one signal often leaves a gap: a metric may identify a symptom but not a request; a log may show one failure but not its prevalence; a trace may show latency but not a long-term trend.

Metrics for aggregate behavior

Counters increase as events occur, such as requests or errors. Gauges represent a value at a point in time, such as active connections. Histograms or summaries record distributions such as latency, allowing percentile or bucket analysis. Choose measurements that correspond to user-visible behavior: request rate, error rate, and latency are common service signals, while queue age and saturation expose capacity constraints.

Metric labels create time series. Keep label cardinality bounded. Labels such as method, route template, status class, and region are often manageable; user id, request id, or an arbitrary URL can create millions of series and large cost. Put high-cardinality identifiers in sampled logs or traces instead. Define units, names, and aggregation semantics consistently so dashboard comparisons remain meaningful.

Logs for event detail

Structured logs use stable fields such as timestamp, severity, service, event name, trace id, and safe contextual attributes. This allows queries without parsing arbitrary prose. Log at meaningful boundaries: request accepted, dependency failed, state transition completed. Avoid logging every internal step at high volume; useful logs are intentional and actionable.

Treat logs as sensitive data. Do not record credentials, tokens, raw personal data, or full request bodies by default. Redact or tokenize necessary identifiers, restrict access, set retention based on operational and regulatory needs, and audit access. A correlation or trace id can connect a record to a request without exposing the person behind it. Define sampling or rate limits for repetitive errors so a failure burst does not exhaust storage.

Traces for request paths

A trace consists of spans with start and end times, parent-child relationships, attributes, and status. A span can represent an inbound request, database call, queue publish, or worker operation. Parent-child structure shows synchronous work; asynchronous handoffs may require links or propagated context to preserve causality across time. Traces are particularly useful for distributed latency because they show which dependency contributes to the end-to-end duration.

Trace data is sampled because recording every span can be expensive. Head-based sampling makes a decision near trace start and is simple, but can miss rare errors. Tail-based sampling evaluates completed traces and can retain errors or slow requests, but requires buffering and central processing. Sampling policy should preserve enough context for critical workflows and comply with data policies. Avoid assuming an unsampled trace means an operation did not occur.

Connect signals without duplicating them

A trace id in a structured log links an event to the request path. Metric exemplars can point from an aggregate latency bucket to a representative trace. Dashboards can link a service error-rate spike to logs and traces for the same interval. Use consistent service names, environment labels, and time semantics across the stack. This creates a diagnostic path without copying every log field into metric labels.

Instrument at boundaries where a signal changes an operational decision. Define who responds to alerts, what healthy ranges look like, and how to investigate. Observability is not a volume contest: a small set of reliable, low-cardinality metrics, useful structured logs, and sampled traces often outperforms an ungoverned firehose.

Diagnose a checkout latency incident

Assume checkout p95 latency rises from 300 milliseconds to 2 seconds while request volume remains steady. Start with a latency histogram and error-rate metric segmented by service, route template, and region. A dashboard shows the increase began after a deployment and is isolated to one region. This narrows the scope without creating a metric label for every customer or request id.

Open a representative slow trace. It shows the checkout API spends most of its time waiting for an inventory call, while database and payment spans remain within baseline. The trace includes the upstream service name, duration, status, and safe request attributes. It does not include a customer's email or authorization token. Follow the linked structured log using the trace id and find a retry-timeout warning. A single trace is an example; use metrics to confirm how widespread the behavior is.

Build the signal path

Instrument the inbound checkout request with a server span and record route template, status, and service version. Create child spans for inventory and payment calls. Emit a counter for total and failed requests and a latency histogram with defined buckets. Keep labels low-cardinality. Log the retry event with a trace id, dependency name, attempt number, and timeout class. Never put the raw order id in a metric label.

Create an alert based on a user-facing SLI, such as successful checkout ratio or latency objective over a rolling window. Page when the burn rate indicates a sustained or fast reliability-budget impact, not for every short fluctuation. Include a runbook link that starts with the service dashboard and dependency trace view. A metric alert tells the on-call engineer where to begin; the trace and logs provide request-level evidence.

Verify sampling and privacy

During normal traffic, head-sample a representative fraction of traces to control cost. Preserve error and unusually slow traces if a tail-sampling system is available. Compare sampled trace counts with request metrics so operators understand what is and is not visible. Document that missing trace data can result from sampling, instrumentation gaps, or propagation loss.

Review logs for sensitive fields and define retention. A trace may carry baggage or attributes from an upstream caller; allowlist what is accepted and prevent untrusted input from injecting high-cardinality or sensitive data. Restrict access to detailed traces and audit queries. If a support investigation needs an order identifier, use a controlled mapping or redacted lookup rather than exposing the identifier to every telemetry consumer.

Exercise the workflow

Inject a slow inventory response in a test environment. Confirm the latency metric crosses the intended threshold, the alert reaches the owning team, a trace shows the dependency wait, and a structured log explains the retry. Then create a failure that is sampled out under the normal policy and verify whether the system has an error-retention mechanism. This makes sampling behavior explicit rather than surprising during an incident.

Finally, check dashboard cost and cardinality after the test. Remove labels that create a new series per user or URL. Confirm service and environment names match across logs, metrics, and traces. A usable observability path links aggregate symptom to representative request evidence while respecting cost, privacy, and operational ownership.

Balance detail, cost, and privacy

Metrics are compact and useful for trends, but aggregation loses individual context. Logs preserve events but can grow rapidly and expose sensitive data. Traces reveal cross-service timing, but sampling can omit requests and span attributes can create privacy or cardinality risks. Design each signal for a diagnostic question and connect them with controlled identifiers rather than duplicating all detail everywhere.

Cardinality is a frequent operational failure. User ids, full URLs, exception messages, or unbounded tenant values in metric labels can overwhelm a time-series backend. Keep bounded dimensions in metrics and move request-level detail to access-controlled logs or traces. Define instrumentation libraries and naming conventions so each team does not invent incompatible labels.

Measure what users experience

Infrastructure health does not guarantee a healthy user journey. A database can be up while checkout is too slow, and a low CPU graph can coexist with a saturated downstream quota. Define service indicators around successful requests, latency, and freshness where appropriate. Alert on symptoms tied to a reliability objective, then use internal signals to diagnose the cause.

Choose trace sampling based on both cost and failure visibility. Head sampling is efficient but may miss a later error; tail sampling can retain slow or failed traces but requires buffering and extra infrastructure. Make sampling policy visible in the tooling and do not interpret absent traces as evidence that no request occurred. Use metrics and logs to determine volume and failure prevalence.

Governance and operations

Telemetry pipelines need ownership, retention, access control, and cost budgets. Redact secrets before export, restrict raw event access, and audit searches. Define who maintains dashboards and alerts, who responds, and how signal changes are reviewed. A dashboard without a decision or owner becomes an expensive archive.

In an interview, map a diagnostic question to each signal: metrics for aggregate health, logs for discrete event context, traces for a distributed request path. Explain how trace ids connect logs, why metric cardinality must be bounded, and how sampling balances cost and rare-failure visibility. Finish with a user-facing SLI and an alert-to-runbook path. This demonstrates observability as an operating capability, not simply three data products.

Retention should match the diagnostic and compliance need. High-volume debug logs may need a short window, while a compact SLI history may support longer reliability reporting. Define deletion and access rules for raw events and sampled traces. Keep an approved process for temporary diagnostic verbosity with an owner and automatic expiry so incident debugging does not become permanent high-cost collection.

Telemetry changes need release discipline. Adding a label can multiply metric series, changing a service name can break dashboards, and removing a field can invalidate queries. Review instrumentation changes with the service owner and platform team. Test dashboards and alerts against representative data before rollout. Track ingest volume and cardinality as system indicators for the observability platform itself.

Alerts should correspond to an action. A page based only on an internal CPU threshold can wake an operator without customer impact, while a user-facing latency objective may be breached before CPU looks unusual. Pair a symptom alert with diagnostic links and a runbook. Use dashboards for exploration, alerts for conditions requiring response, and logs or traces for detailed investigation. This separation reduces alert fatigue and helps teams know which signal to trust first.

In an interview, describe a real diagnostic path from an aggregate metric to a representative trace and structured event. Explain sampling gaps and privacy controls. State how cardinality, retention, and ownership affect cost. Observability succeeds when the signal leads to an operational decision and the data remains safe and useful over time.