Metrics, Logs, and Traces

Three questions arrive in this order during an incident. Is the service healthy at all — a yes/no that a monitoring system must answer continuously. Is it slower than usual, and for which endpoint — a quantitative comparison over time. What happened to this specific request — a narrative only the request itself can tell. Metrics, logs, and traces exist because no single record type answers all three cheaply: a metric answers the first two with a number you can graph, a log answers the third with narrative, and a trace answers the third with timing across services.

The first instrument a service ships is not a graph — it is the health endpoints, whose two flavors have genuinely different semantics:

var ready atomic.Bool // flipped false at shutdown start, true after init

func healthRouter() http.Handler {
	mux := http.NewServeMux()

	// Liveness: is this process functional? Restart on failure.
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "ok")
	})

	// Readiness: should traffic be routed here? Fail during drain.
	mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
		if !ready.Load() {
			http.Error(w, "draining", http.StatusServiceUnavailable)
			return
		}
		fmt.Fprintln(w, "ok")
	})
	return mux
}

Conflating them produces one of two classic failures. Returning 503 from liveness when a dependency is slow causes restart storms: the process is fine, the restart helps nothing, and the churn makes the dependency worse. Returning 200 from readiness during shutdown keeps the load balancer sending traffic into a closing listener — exactly the dropped-request scenario Graceful Shutdown and Connection Drain exists to prevent. The rule: liveness measures self, readiness measures usefulness to the router, and readiness is the one that flips early.

With health covered, the RED framework frames instrumentation: for every endpoint, measure Rate (requests/sec), Errors (failures/sec), Duration (latency distribution). Those three numbers localize most incidents — rate says it is traffic, errors say it is correctness, duration says it is a dependency or resource. Each has a natural Go representation: a counter for rate and errors, a histogram for duration. The framework's premise is worth stating because it justifies the layout: you instrument the service boundary first, then descend to internals only when the boundary numbers demand it — the same narrowing an incident investigation performs as it moves from dashboards to a single request.

What makes instrumentation durable is wiring it into the middleware layer once, rather than scattering increments through handlers. The next chapter builds that middleware; the principle now: every measurement needs a definition someone else can interpret at 3 a.m. — p99 of what, errors by which class, over which window.

The metrics endpoint works. Then someone adds user_id as a label to the request counter, and within a day the metric holds 2 million series, the Prometheus scrape takes 40 seconds, and the query that drew a tidy latency graph now times out. This is the cardinality tax, and it is the single most common way a metrics system dies. Labels are dimensions for grouping; every distinct label combination creates a new time series that must be stored, indexed, and evaluated forever.

The RED instrumentation for an HTTP service, written once as middleware:

var (
	reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "http_request_duration_seconds",
		Buckets: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5},
	}, []string{"method", "route", "code"})
	reqTotal = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "http_requests_total",
	}, []string{"method", "route", "code"})
)

func metrics(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		start := time.Now()
		rw := &statusWriter{ResponseWriter: w, status: http.StatusOK}
		next.ServeHTTP(rw, r)
		dur := time.Since(start)

		route := routeTemplate(r) // "/users/{id}", NOT "/users/8821"
		code := strconv.Itoa(rw.status)
		reqDuration.WithLabelValues(r.Method, route, code).Observe(dur.Seconds())
		reqTotal.WithLabelValues(r.Method, route, code).Inc()
	})
}

Three decisions carry the cost and the value:

Label by route template, not raw path. /users/8821, /users/8822, … a raw path label creates a series per user id — unbounded. The label must be the pattern (/users/{id}), which requires routing middleware that knows the matched template (Go 1.22's r.PathValue makes the pattern available; otherwise derive it from your router). This one rule is the difference between 20 series and 2 million.

Buckets chosen for the service's latency shape, not defaults. Histogram buckets determine the resolution of your percentile estimates: if 99% of requests take 30 ms, buckets that jump from 100 ms to 250 ms make p99 a fiction. Start from what users feel (single-digit ms to a few seconds), and note that buckets are upper bounds — a 50 ms bucket counts anything ≤ 50 ms, so finer resolution near the alert threshold pays for itself.

Errors as a label, not a separate metric. code (or a bounded outcome of success/retry/fail) lets one query slice errors by route and status class. What must not become a label: user ids, URLs with query strings, exception messages, or any value with unbounded or high-cardinality domains. The test for a label: can you enumerate its possible values? Method, route, status, region, instance — enumerable. Request id, user id, error text — never.

This is where metrics and logs split responsibilities cleanly. The metrics answer which route got slower and by how much; they cannot answer why this one request took four seconds — the error message, the upstream host, the retry count. That narrative belongs to structured logging with correlation fields, which the next chapter ties to the same request. The discipline: put counts and distributions in metrics with bounded labels, put narrative in logs, and never try to smuggle one into the other's job.

The dashboard says checkout p99 rose from 400 ms to 4 s at 14:02. Now what? Without correlation, the next hour is spent guessing which log lines belong to the slow requests — grepping timestamps and hoping the ids in the message text line up. Correlation is the mechanism that turns three separate signals into one investigation: a request id (or trace id) that appears in the log records, labels or exemplars on the metrics, and the span tree of the trace.

The mechanism in Go is context, established once at the edge:

type ctxKey struct{}

func withRequestID(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		id := r.Header.Get("X-Request-Id")
		if id == "" {
			id = newID()
		}
		w.Header().Set("X-Request-Id", id)

		l := slog.With("request_id", id, "route", routeTemplate(r))
		ctx := context.WithValue(r.Context(), ctxKey{}, l)
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

// anywhere downstream
func (s *Service) charge(ctx context.Context, amt int) error {
	loggerFrom(ctx).Info("charging", "amount_cents", amt)
	// ...and propagate the id to the next service:
	req, _ := http.NewRequestWithContext(ctx, "POST", bankURL, body)
	req.Header.Set("X-Request-Id", requestIDFrom(ctx))
	return doPost(req)
}

A logger carried in the context means every line from one request shares request_id without business code naming it — the pattern Structured Logging with log/slog established, now closed into a loop at the edge. The id must also travel outbound: forwarding X-Request-Id (or the W3C traceparent header) to downstream services is what makes one request traceable across process boundaries. An id that stops at your service documents only your half of the story.

Traces add what logs structurally cannot: parent-child timing across services. A trace is a tree of spans — checkout → payments → inventory, each with start and duration — so "slow because inventory" becomes a visual fact rather than an inference. In Go, the OpenTelemetry SDK provides the API, and instrumentation usually arrives as middleware and otelhttp wrappers rather than hand-rolled span code; the discipline that makes traces useful is manual: meaningful span names (GET /users/{id} not http handler), attributes for the fields you debug by (route, status, upstream host), and links between the trace id and the log records emitted inside the span. The payoff of a well-formed trace is exemplars — a metric point that links to the trace of a representative request — which lets you jump from "this histogram bucket got heavier" to "here is a request inside it."

Alert on symptoms, then descend. The alerting layer watches what users feel — error rate and latency SLOs from the RED metrics — because those pages a human. The investigation layer uses the trace to find the slow span, the logs inside that span for the error text, and the metrics again to scope the blast radius (one route? one instance? all regions?). This flow — metric tells you that, trace tells you where, log tells you why — is the entire architecture of the three signals, and a service that answers all three questions from its own instrumentation can be operated by someone who has never read its code.