Prometheus for Docker

Exposing Metrics

Prometheus pulls, so something must serve a /metrics text endpoint. Applications instrument with a client library:

import { Registry, Counter } from "prom-client";
const registry = new Registry();
registry.registerMetric(new Counter({
  name: "http_requests_total",
  help: "HTTP requests",
  labelNames: ["method", "route", "status"],
}));

Infrastructure metrics come from exporters: cAdvisor reports per-container CPU, memory, and network from cgroups; node_exporter covers the host. Run both as containers with their ports published, and the same scrape job picks them up as it does your app.

Scrape Configuration

Containers change IPs, so scrape published ports or file-based targets:

scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["host.docker.internal:3000"]
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]
  - job_name: docker-labels
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.json"]

Run Prometheus on the same Compose network as exporters (cadvisor:8080 resolves by service name) and mount its config read-only. file_sd lets a script or discovery step rewrite target JSON without touching prometheus.yml, which keeps reloads declarative. Verify with the /targets page: each target must be UP, not merely listed.

Pitfalls and Interview Angles

Ephemeral IPs. Scraping container-ip:port breaks the moment the container is recreated; scrape stable names (service DNS, published ports) or regenerate file_sd files on change.

Cardinality. Label values like request IDs or user IDs multiply time series and can exhaust memory; label with bounded dimensions (route, status) and log the rest.

Pull surprises. If a container has no published port and Prometheus is not on its network, the scrape silently fails — network reachability is part of the metric design.

Interview: describe the pull model, name the three exporter layers (app, cAdvisor, node_exporter), and explain how you would keep targets correct across container recreates.