Prometheus Monitoring

The Pull Model and Time Series

Prometheus scrapes an HTTP /metrics endpoint from every target every scrape_interval (default 1m) and stores samples in a local TSDB. A series is identified by metric name plus its full label set:

http_requests_total{job="api", code="500", method="GET"}

On Kubernetes it discovers targets dynamically through kubernetes_sd_configs (role: pod, endpoints, service), so no static target list exists. PromQL reads historical ranges; the push model (Pushgateway) is only for short-lived jobs. Retention defaults to 15 days locally — long-term storage is a separate concern.

Deploying and Querying

The kube-prometheus-stack Helm chart installs the Prometheus Operator, which turns ServiceMonitor CRDs into scrape configs — pods are selected by the labels on their Services:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
spec:
  selector:
    matchLabels: { app: api }
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

Core PromQL:

rate(http_requests_total{code=~"5.."}[5m])
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

rate handles counter resets; le buckets are what make percentile math possible on a histogram.

Pitfalls and Interview Angles

Cardinality explosion. Labels like user_id or request_id create millions of series and blow up memory — keep labels to low, bounded dimensions (route, code, region).

Wrong range selectors. rate needs a window at least 4× the scrape interval; querying a counter directly shows raw totals, not speed, and graphs break on restarts without rate.

Silent staleness. A target that stops responding disappears after the staleness window — absence of a series is itself a signal worth alerting on.

Interview angle: explain pull vs push and why discovery fits dynamic pods, derive a p95 from histogram buckets, and defend your label schema against cardinality.