cAdvisor & Metrics

Why Container Metrics Differ

Host tools like top average CPU across every container on a node, so one runaway pod hides inside a healthy-looking number. Scraping cgroup counters by hand per node works for one machine and collapses at fifty.

cAdvisor (Container Advisor), built at Google, solves the per-node half: it discovers containers automatically, samples cpu, memory, filesystem, and network usage from cgroups, and serves the results over HTTP.

kubectl top pod web-6f8d4c-x2n9l   # pod CPU/memory sourced from kubelet's cAdvisor

In Kubernetes the kubelet embeds cAdvisor, so the summary API and metrics-server consume the same sampler without installing an agent.

Scraping the Numbers

Modern clusters expose cAdvisor data through the kubelet on port 10250 rather than a separate cAdvisor port (the 4194 UI was removed in 1.12):

curl -sk https://node1:10250/stats/summary | jq '.pods[0]'
curl -sk https://node1:10250/metrics/cadvisor | grep container_cpu

Each record carries cpu.usage, memory.usage, and filesystem I/O counters keyed by pod and container name. Two consumers read them differently: metrics-server keeps a short rolling window to compute the rates behind kubectl top, while Prometheus scrapes container_cpu_usage_seconds_total and stores history for dashboards and alerts.

Gaps and Interview Angles

Pitfall 1 — treating metrics-server as monitoring. It stores only the latest scrape in memory, so kubectl top shows the present moment and nothing historical. Ask it for last week's CPU and it has no answer; Prometheus handles that job.

Pitfall 2 — alerting on the wrong memory number. container_memory_usage_total counts reclaimable page cache, so a cache-heavy app looks like a leak. Alert on container_memory_working_set_bytes, the figure kubelet uses for OOM decisions.

Interview-ready: state that kubelet embeds cAdvisor, name one counter per resource, and explain the split between in-memory metrics-server and a real time-series store.