The Single Front Door
The Single Front Door
kube-apiserver is the cluster's only public surface. kubectl, controllers, kubelet, and the scheduler all speak REST to it; direct etcd access is reserved for backup tooling.
Its responsibilities stack in order:
- Authentication — certificate, bearer token, or OIDC identity.
- Authorization — RBAC decides whether this verb on this resource is allowed.
- Admission — validate and mutate the object.
- Persistence — the sole writer to etcd.
- Watch — clients subscribe to changes instead of polling.
That watch API is the nervous system. The Deployment controller does not poll every second; it receives an event the instant an object changes. Cluster scale is largely the scale of these long-lived connections.
etcd Storage and Snapshots
etcd Storage and Snapshots
etcd is a Raft-replicated key-value store: writes need acknowledgement from a quorum (2 of 3, 3 of 5), so write latency tracks the slowest disk in the majority.
Objects live under keys such as /registry/pods/default/web-abc123. One implication: Secrets sit there as base64 unless encryption at rest is configured.
ETCDCTL_API=3 etcdctl snapshot save snap.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
etcdctl snapshot status snap.db -w table
Schedule auto-compaction (for example --auto-compaction-retention=8) and watch disk before history outgrows the volume.
API Server Under Load
API Server Under Load
Pitfall 1 — naive list loops. Code that lists every pod every few seconds scales cost with cluster size. Use resourceVersion=0 cached reads or informers (watch-based) instead.
Pitfall 2 — oversized objects and unbounded watches. A single etcd value caps at about 1 MiB; giant ConfigMaps plus thousands of watchers inflate apiserver and etcd memory alike.
Interview-ready: describe the write path — auth, RBAC, admission, etcd, watch event — then answer "apiserver latency spiked": check apiserver_request_duration_seconds, audit-log volume, and etcd fsync latency. Note that clients back off on 429/5xx and kubelet retries, so a brief stall degrades control, not running workloads.
Quiz
1. Where do controllers receive change notifications?
2. What is the practical size limit for a single object in etcd?
3. A cluster with 3 etcd members loses one. What happens?
Flashcards
Question
Why only apiserver writes etcd
Click to reveal answer
Answer
Centralizes auth, admission, and validation so storage invariants hold cluster-wide.
Question
Watch vs poll
Click to reveal answer
Answer
Watch streams change events from apiserver; polling lists objects on an interval and wastes QPS.
Question
etcd snapshot command
Click to reveal answer
Answer
etcdctl snapshot save with endpoints and TLS certs, then snapshot status to verify.
Question
First checks for apiserver slowness
Click to reveal answer
Answer
apiserver_request_duration_seconds, audit volume, etcd fsync latency, client retries on 429.
Revision Notes
Key Takeaways
- 1. All cluster traffic converges on kube-apiserver: auth, RBAC, admission, persistence
- 2. etcd quorum and disk latency govern write performance
- 3. Watches scale better than repeated lists; size objects small
Interview Tips
- • Recite the write path end to end, including admission plugins
- • For an outage scenario, separate control-plane impact from workload impact
Cheat Sheet
Path: auth → RBAC → admission → etcd → watch event. Quorum writes need majority; snapshot etcd before upgrades; cap object size and prefer watches over list loops.