Logging Fundamentals

The Container Logging Model

Kubernetes treats stdout and stderr as the log streams: the container runtime writes them to files on the node, and the kubelet manages them. Nothing is shipped anywhere by default — logs exist only as long as the pod (and node) do.

Kubelet rotation flags keep disks bounded:

--container-log-max-size=10Mi   # rotate after this size
--container-log-max-files=5     # keep this many rotated files

Applications may also write to files inside the container, but then nothing rotates them and kubectl logs cannot see them — prefer stdout unless a framework demands a file.

Reading Logs with kubectl

kubectl logs deploy/api -f --tail=200        # stream the latest lines
kubectl logs api-7d9f -c sidecar             # pick a container
kubectl logs api-7d9f -p                     # previous (crashed) container
kubectl logs api-7d9f --since=15m --timestamps

On the node, raw files sit under /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log with symlinks in /var/log/containers/. kubectl logs does not query etcd or the API server — the request is proxied to the kubelet, which tails those files. Multi-container pods require -c; without it you get an error whenever the pod has more than one container.

Pitfalls and Interview Angles

Node loss = log loss. Logs are node-local. When a pod is evicted or the node dies, its logs go with it unless something shipped them first. That is the entire rationale for agents like Fluent Bit running as DaemonSets.

Crashed pods need -p. Once the container restarts, live logs start over; --previous shows the output before the crash.

Timestamps missing. Application timestamps may be absent or in another timezone — use --timestamps and rely on kubelet-prefixed RFC3339 lines when correlating.

Interview angle: name the full path (runtime writes file → kubelet serves kubectl logs), explain why logs are not stored in etcd, and say what you add to survive node failure.