Reliability Prerequisites
Reliability Prerequisites
Before traffic, confirm the workload layer can survive routine disruption:
- Multiple replicas across nodes/zones with pod anti-affinity or topology spread — a single replica is a single point of failure
- PodDisruptionBudgets so
kubectl draincannot take out an entire service:kubectl get pdb -A - Requests/limits on every container (no BestEffort strays): check with a policy or
kubectl get pods -A -o jsonreview - Probes configured — readiness for traffic gating, liveness only for hangs
- Autoscaling headroom — HPA bounds set, node autoscaler healthy, capacity alerts firing before saturation
Workloads without these fail the first node upgrade or rolling restart you attempt.
Security Prerequisites
Security Prerequisites
- Least-privilege RBAC — review bindings, and probe real identities:
kubectl auth can-i list secrets --as=system:serviceaccount:prod:api -n prod - Pod Security Admission enforced per namespace (baseline minimum, restricted where possible)
- No privileged pods / hostPath by policy — enforce with admission (PSA, Kyverno, or OPA)
- Image hygiene — digest-pinned images, scanning in CI, admission requiring trusted registries
- Secrets hygiene — encryption at rest via a KMS provider, no secrets in Git, rotation procedure documented
- NetworkPolicy — default-deny in sensitive namespaces with explicit allows
Every item here maps to an object you can list; anything unverifiable by command is a claim, not a control.
Operability and Verification
Operability and Verification
- Observability: metrics scraped (ServiceMonitors present), logs shipped centrally, alerts on saturation and error rate with tested routes
- Backups: Velero schedules running and restored at least quarterly with recorded RTO
- Certificate/component expiry: watch kubelet/client certs and Ingress certs —
kubeadm certs check-expirationon kubeadm clusters - Capacity: node headroom alerts, PDB-safe drain procedure documented and rehearsed
- Change safety: GitOps or CI with diffs, rollback path known (
helm rollback/rollout undo), runbooks linked to alerts
Interview framing: present the checklist as layers — workload → namespace → cluster → supply chain — and for each item name the command that proves it. Prioritization rule: anything that turns a small failure into a cluster-wide outage outranks convenience features.
Quiz
1. Which item protects against a node drain removing all replicas of a service?
2. How do you verify what a ServiceAccount can actually do?
3. Which checklist item is only proven by performing it?
Flashcards
Question
Reliability core five
Click to reveal answer
Answer
Multi-replica + spread, PDBs, requests/limits, probes, autoscaling headroom.
Question
kubectl auth can-i
Click to reveal answer
Answer
Probes effective RBAC for an identity (--as=system:serviceaccount:ns:name); --list shows full permission set.
Question
Drain safety
Click to reveal answer
Answer
kubectl drain --ignore-daemonsets --delete-emptydir-data with PDBs present keeps services available during maintenance.
Question
Proven backup
Click to reveal answer
Answer
A scheduled backup plus a timed restore drill — only the drill establishes real RTO.
Revision Notes
Key Takeaways
- 1. Checklist layers: workload → namespace → cluster → supply chain
- 2. Every claim needs a proving command; unverifiable items are debt
- 3. Prioritize against blast radius: outage amplifiers before conveniences
Interview Tips
- • Present the checklist layered, and cite the verification command for each item you name
- • Pick the three highest-risk gaps you would fix first in a greenfield cluster, with reasoning
Cheat Sheet
Pre-prod: replicas+spread, PDBs, requests/limits, probes, HPA; RBAC least-priv (auth can-i), PSA, no privileged, digest images, KMS secrets; NetPol default-deny; alerts, tested restores.