Alerting with Alertmanager

From Rule to Firing Alert

Prometheus evaluates rule groups every evaluation_interval. An alert expression starts in pending, becomes firing only after it holds true for the rule's for: duration, then ships to Alertmanager.

groups:
- name: api
  rules:
  - alert: ApiHighErrorRate
    expr: |
      sum(rate(http_requests_total{code=~"5.."}[5m]))
        / sum(rate(http_requests_total[5m])) > 0.05
    for: 10m
    labels: { severity: critical }
    annotations:
      summary: "5xx ratio above 5% for 10 minutes"

for: exists to absorb blips — without it, a one-sample spike pages someone. Labels like severity and team become the routing keys downstream.

Routing in Alertmanager

Alertmanager deduplicates, groups, and routes each alert down a tree of matchers:

route:
  group_by: [alertname, cluster]
  group_wait: 30s
  repeat_interval: 4h
  routes:
  - matchers: [severity="critical"]
    receiver: pagerduty
    continue: true
  - receiver: slack
receivers:
- name: pagerduty
  pagerduty_configs: [{ service_key: XYZ }]
- name: slack
  slack_configs: [{ api_url: hooks.slack.com/..., channel: "#ops" }]

First matching child wins unless continue: true. Inhibition silences a warning while its cause fires (suppress node-down warnings for alerts on that node); silences mute known maintenance windows via amtool or the UI with an expiry.

Pitfalls and Interview Angles

Alert symptoms, not causes. Page on user-visible error rate; ticket on disk filling. Cause-based pages (CPU, pod restarts) without user impact train on-call to ignore everything.

No for: window → flap pages on transient GC pauses or scrape gaps.

Routing surprises. group_by on high-cardinality labels bursts notifications; a matcher using =~ on a missing label silently drops alerts to the default receiver.

Dead channels. Keep a watchdog alert that fires forever — if it stops arriving, the pipeline itself is broken.

Interview angle: walk a critical alert end-to-end (rule → pending → firing → route → receiver), then explain how inhibition stops a node failure from generating twenty pages.