Monitoring Design

The Three Pillars of Observability

Observability is the ability to understand the internal state of a system by examining its outputs. The three pillars are:

• Logs: Discrete, timestamped events that record what happened
• Metrics: Numerical measurements aggregated over time
• Traces: End-to-end request flows across service boundaries

Together these give you full visibility. Logs tell you what went wrong, metrics tell you when and how much, and traces tell you where in the chain.

Structured Logging

Unstructured logs like User 123 logged in at 3pm are hard to query. Structured logs use a consistent format (usually JSON):

{
  "timestamp": "2026-08-16T14:23:01.123Z",
  "level": "INFO",
  "service": "auth-service",
  "traceId": "abc-123-def-456",
  "spanId": "span-789",
  "message": "User authenticated successfully",
  "userId": "user_42",
  "method": "POST",
  "path": "/api/login",
  "durationMs": 145,
  "statusCode": 200
}

Why structured logging matters:

• Enables query by field: service:auth-service AND level:ERROR
• Facilitates aggregation: average durationMs per endpoint
• Supports correlation across services via traceId
• Machine-parseable for automated alerting

Correlation IDs

A correlation ID (or trace ID) is a unique identifier assigned to a request at the edge of your system. Every log entry across all services includes this ID, allowing you to reconstruct the full journey of a single request.

API Gateway generates traceId: abc-123
  -> Auth Service logs: { traceId: abc-123, message: "Token validated" }
  -> Order Service logs: { traceId: abc-123, message: "Order created" }
  -> Payment Service logs: { traceId: abc-123, message: "Charge successful" }

Without correlation IDs, debugging a failing request across 5 microservices is nearly impossible.

Log Levels

Level	When to Use	Example
ERROR	Something broke, needs immediate attention	Database connection failed
WARN	Something unexpected but not broken	Retry attempt 2 of 3
INFO	Normal business events	Order placed
DEBUG	Detailed diagnostic info	Query result: 42 rows

Best practice: Production should run at INFO level. Enable DEBUG only when troubleshooting specific issues.

Log Retention and Storage

• Hot storage (0-7 days): Fast query, expensive (e.g., Elasticsearch)
• Warm storage (7-30 days): Slower query, cheaper (e.g., S3 with Athena)
• Cold storage (30+ days): Archive, cheapest (e.g., S3 Glacier)

Cost grows linearly with log volume. A service processing 10K requests/sec generates ~1TB of logs per day. Retention policies are critical for cost control.

The RED Method

The RED method is a framework for monitoring microservices, focused on the request-driven nature of services:

• Rate: Number of requests per second (traffic volume)
• Errors: Number of failed requests per second (error rate)
• Duration: Distribution of request latencies (latency)

These three metrics capture the most important signals. If a service is healthy, these metrics look normal. If any of them deviates, you have a problem.

The USE Method

For infrastructure (servers, databases, queues), use the USE method:

• Utilization: Percentage of resource in use (CPU at 85%)
• Saturation: Queue depth or backlog (1000 pending requests)
• Errors: Count of error events (disk read errors)

Key Infrastructure Metrics

Resource	Utilization	Saturation	Errors
CPU	% usage	Run queue length	Hardware errors
Memory	% used	Swap usage	OOM kills
Disk	IOPS usage	I/O wait queue	Read/write errors
Network	Bandwidth usage	TCP retransmits	Packet drops

Prometheus + Grafana

Prometheus scrapes metrics from instrumented services at regular intervals (e.g., every 15s). Metrics are stored in a time-series database.

Service instrumentation (Go example):

import "github.com/prometheus/client_golang/prometheus"

var requestDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name:    "http_request_duration_seconds",
        Help:    "Duration of HTTP requests",
        Buckets: []float64{.01, .05, .1, .25, .5, 1, 2.5, 5, 10},
    },
    []string{"method", "path", "status"},
)

// In handler:
requestDuration.WithLabelValues("POST", "/api/orders", "200").Observe(duration.Seconds())

PromQL queries:

# 99th percentile latency over 5 minutes
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

# Error rate: 5xx responses / total responses
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

# Requests per second by service
sum(rate(http_requests_total[5m])) by (service)

Grafana provides dashboards. Key panels:

• Request rate over time
• Error rate percentage
• Latency distribution (p50, p95, p99)
• Saturation indicators (queue depth, connection pool usage)

Alerting

Alert rules in Prometheus:

groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for 5 minutes"

Escalation pipeline:

1. Alert fires in Prometheus/AlertManager
2. PagerDuty receives alert, pages on-call engineer
3. If unacknowledged in 15 minutes, escalates to secondary
4. If unacknowledged in 30 minutes, escalates to engineering manager

On-call rotation: Engineers rotate weekly. Each week has a primary and secondary on-call. Handoff includes a written summary of ongoing issues.

Health Checks

Liveness probe: "Is the process alive?" If this fails, the orchestrator (Kubernetes) restarts the container. Use for detecting deadlocks or infinite loops.

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 3

Readiness probe: "Can the process accept traffic?" If this fails, the orchestrator removes the pod from the load balancer but does NOT restart it. Use for detecting dependency failures (database down, cache cold).

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 3
  failureThreshold: 2

Critical difference: If your readiness probe checks database connectivity and the DB is temporarily down, you don't want to restart every pod (that would make things worse). Readiness fails, traffic stops, pods stay alive and recover when DB returns.

Why Distributed Tracing?

In a monolith, a stack trace tells you what went wrong. In microservices, a single user request may touch 10-20 services. You need to track the request across all of them.

A distributed trace is a tree of spans. Each span represents one unit of work (an HTTP call, a DB query, a cache lookup).

[API Gateway] traceId=abc-123
  |-- [Auth Service] spanId=span-1, duration=15ms
  |-- [User Service] spanId=span-2, duration=45ms
       |-- [PostgreSQL] spanId=span-3, duration=30ms
  |-- [Order Service] spanId=span-4, duration=120ms
       |-- [Redis] spanId=span-5, duration=2ms
       |-- [Payment Service] spanId=span-6, duration=80ms
       |-- [Inventory Service] spanId=span-7, duration=25ms

From this trace, you can see the order service took 120ms total, with 80ms spent waiting for payment and 2ms for cache.

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for instrumentation. It provides:

• APIs for generating traces, metrics, and logs
• SDKs for auto-instrumentation in most languages
• Exporters to send data to backends (Jaeger, Zipkin, Datadog, AWS X-Ray)

Auto-instrumentation (Python example):

from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="otel-collector:4317"))
provider.add_span_processor(processor)

# Auto-instrument Flask
FlaskInstrumentor().instrument_app(app)

Manual span creation:

from opentelemetry import trace

tracer = trace.get_tracer("order-service")

with tracer.start_as_current_span("process-payment") as span:
    span.set_attribute("payment.method", "credit_card")
    span.set_attribute("payment.amount", 99.99)
    # ... payment logic

Trace Backends

Backend	Type	Best For
Jaeger	OSS, self-hosted	Full control, CNCF native
AWS X-Ray	Managed	AWS-centric workloads
Datadog APM	SaaS	All-in-one observability
Honeycomb	SaaS	High-cardinality analysis

Sampling Strategies

Tracing generates enormous volume. Sampling controls what you keep:

• Head-based: Decide at the start of a request. Simple, but may miss interesting traces.
• Tail-based: Collect all traces, decide later which to keep. Higher fidelity, but requires more storage.
• Adaptive: Sample more aggressively during normal traffic, less during anomalies.

Production tip: Sample 1% of normal requests, 100% of error requests.

Real-World Monitoring Architecture

For a microservices architecture serving 10K req/sec:

1. Instrumentation layer: OpenTelemetry SDK in each service
2. Collection layer: OTel Collector (agent per node) + OTel Collector (gateway)
3. Storage layer: Prometheus (metrics), Elasticsearch (logs), Jaeger (traces)
4. Visualization: Grafana (metrics), Kibana (logs), Jaeger UI (traces)
5. Alerting: Prometheus AlertManager -> PagerDuty
6. Correlation: All three pillars share traceId, enabling cross-signal investigation

Cost estimation (rough): At 10K req/sec, expect ~$2-5K/month for observability infrastructure (excluding SaaS tools).