Observability for Production GraphQL APIs

HTTP 200 Is Not Health

The dashboards for a REST service map cleanly onto status codes: 4xx is the client, 5xx is you, latency is one number per endpoint. GraphQL collapses all of that into one status code that stays 200 while the service falls apart underneath:

{
  "data": { "product": { "name": "Bottle", "reviews": null } },
  "errors": [{ "path": ["product", "reviews"],
               "extensions": { "code": "SUBGRAPH_ERROR" } }]
}

That response is a 200 with a broken field in it. If your alerting watches status codes, a subgraph can be down for twenty minutes before anyone notices, and the first signal will be a user complaining that reviews are missing. The same blindness applies to latency: an operation that normally takes 80ms can take 2.4s and still be a green 200, because the slowness lives in one nested resolver that no endpoint-level metric sees.

Nullability sharpens the difference between a cosmetic failure and a real one. A null on an optional field is a hole a client can hide behind a placeholder; a null on a non-null field bubbles up and nulls its entire parent, so one failing subgraph can blank a whole product card. Both arrive as a 200, both are invisible in an access log, and error counts that ignore which path went null will tell you something broke without telling you what the user lost.

GraphQL therefore needs four instruments, each answering a question the others cannot:

Start with error taxonomy, because it is the cheapest win: an error counter labelled with extensions.code and a phase (parse, validate, auth, resolve) turns "errors are up" into "auth failures tripled after the token rollout", and those two facts have completely different runbooks.

Instruments That Fit GraphQL

Instrument at three seams: the operation, the resolver, and the subgraph fetch. Each seam has a natural span and a natural metric.

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('graphql-api');
const fieldMs = meter.createHistogram('graphql.field.duration', { unit: 'ms' });
const opErrors = meter.createCounter('graphql.operation.errors');

export function withFieldTiming(parentType, fieldName, resolve) {
  const coord = parentType.name + '.' + fieldName;  // bounded: schema size is bounded
  return (parent, args, ctx, info) => {
    const start = performance.now();
    try {
      const value = resolve(parent, args, ctx, info);
      return value;
    } finally {
      // record on the synchronous path; wrap promises with .finally() when async
      fieldMs.record(performance.now() - start, { field: coord });
    }
  };
}

// errors, labelled by code and phase — never by message text
didEncounterErrors({ errors }) {
  for (const e of errors) {
    opErrors.add(1, {
      code: e.extensions?.code ?? 'UNKNOWN',
      phase: phaseOf(e),          // parse | validate | auth | resolve
      operation: info.operationName ?? 'anonymous',
    });
  }
}

Labels are the design decision. field: "Product.reviews" has a cardinality bounded by your schema — a few thousand at most. query: rawDocument, userId, or traceId are unbounded: each becomes a time series, and the metrics backend either runs out of memory or starts sampling away exactly the data you wanted. Operation name is safe because clients name their operations; anonymous documents need a bucket of their own.

Traces carry the detail metrics cannot. One span per operation, child spans per subgraph fetch and per slow resolver, and trace context propagated into every subgraph so the whole fan-out renders as one tree. That single view answers the question metrics can only hint at: the operation p95 is high because reviews waited 1.8s on a cold connection pool, and you can see it without correlating three dashboards by eye.

Attach operationName and the top-level field list to every log line alongside the request id. When a user says "the product page is slow", one grep on the operation name returns the traces, and one trace returns the subgraph.

Alerts That Name a Runbook

A useful alert tells the on-call engineer which document to open. Four GraphQL-specific signals earn their pager entries:

1. Error-entry rate per operation — count of responses containing errors[] entries, split by phase. A spike in auth means tokens; a spike in resolve means a subgraph or a query shape. This is the alert that replaces "HTTP 5xx", which in GraphQL barely moves.
2. p95 latency per operation name, not per endpoint. There is one endpoint; the operation is the real unit of work, and one pathological document hides inside the aggregate otherwise.
3. Subgraph fetch error and timeout rate, labelled by subgraph. Federation's value is partial failure — the gateway keeps serving when one owner is sick — and that value becomes a silent degradation unless you alert on the per-subgraph series.
4. Rejection counters: cost-budget rejections and persisted-query not-found storms. The first means a client shipped an unbounded document; the second usually means a manifest shipped without the queries clients send. Both fail in ways no 5xx ever shows.

Add plan-cache hit rate as a leading indicator: a sudden drop means new document shapes are arriving at the gateway, which predicts planning latency and often precedes a client release incident.

What not to do. Do not alert on raw error messages (unbounded, and they change with wording); do not page on a single rejected operation; do not build a dashboard whose panels are top-20 queries by raw text — use operation names and field paths so the panels stay stable across releases.

The dashboard that earns its screen space is small: error rate split by phase, p95 by operation, per-subgraph latency and error rate, top ten slowest fields, and the rejection counters. Every panel should answer "which layer do I open?" — validation, auth, resolver, subgraph, or gateway — because that is the only distinction that changes what the on-call does next.