Failure Classes and Targets
Failure Classes and Targets
Separate the scenarios before buying tools:
- Control-plane loss (etcd corruption, quorum loss) — requires etcd snapshots and recovery procedures
- Node/pool loss — handled by replicas + PDBs + autoscaling; usually no restore needed
- Region loss — demands a second cluster and data replication, not just backups
- Accidental deletion — GitOps helps for manifests; app data still needs point-in-time backup
Quantify with RPO (maximum tolerable data loss: RPO 15m ⇒ snapshot at least every 15 minutes) and RTO (time to restore service: affects how much automation your restore path needs). Managed control planes shift etcd duty to the provider but not app-data RPO.
Etcd Snapshots and Restore
Etcd Snapshots and Restore
Take snapshots with etcdctl against a healthy member:
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /backup/etcd-$(date +%F).db
etcdctl snapshot status /backup/etcd-$(date +%F).db -w table
Restore on a clean member: stop the static-pod control plane, run etcdctl snapshot restore ... --data-dir /var/lib/etcd-restore, point the apiserver manifest at the new data dir, and restart — then verify kubectl get nodes and workloads. Pair this with Velero (K8s objects + PVs) and IaC that can rebuild nodes/cluster from scratch. Store copies offsite (3-2-1) and rehearse: an untested restore has an unknown RTO.
Pitfalls and Interview Angles
Pitfalls and Interview Angles
Backups without restores. The graveyard of DR plans: nobody practiced, the runbook references a dead bucket, credentials expired. Measure RTO in drills, quarterly.
File-copying etcd. Copying /var/lib/etcd mid-write yields inconsistent state — always use etcdctl snapshot.
Metadata-only protection. Backing up objects but not PV data restores the shell, not the database; enable volume snapshots or file-level backup for stateful sets.
Single-region copies. A backup in the same region fails the region-outage scenario — replicate.
GitOps illusion. Git reconstructs manifests, not data; say so explicitly when scoping DR.
Interview angle: classify a scenario you are given (control plane vs workload vs region), state RPO/RTO you would propose, and name the exact artifact (snapshot, Velero backup, replicated DB) that satisfies each.
Quiz
1. RPO of 15 minutes means what operationally?
2. Why not copy /var/lib/etcd with cp while the cluster runs?
3. A restore succeeds but the app is missing data. What gap does this show?
Flashcards
Question
RPO vs RTO
Click to reveal answer
Answer
RPO = acceptable data loss (backup frequency); RTO = acceptable time to restore service (automation needs).
Question
etcdctl snapshot
Click to reveal answer
Answer
The only correct way to back up etcd — a consistent API-level copy, verified with snapshot status.
Question
DR layers
Click to reveal answer
Answer
etcd snapshots (control plane) + Velero (objects/volumes) + IaC (infra) + multi-cluster (region loss).
Question
Restore drill
Click to reveal answer
Answer
Timed rehearsal of full recovery; the only way to know actual RTO and broken runbook steps.
Revision Notes
Key Takeaways
- 1. Classify failures first; each layer (etcd, volumes, cluster, region) needs its own artifact
- 2. RPO sets backup cadence, RTO sets how automated the restore must be
- 3. Untested backups are hypotheses — drills convert them into measured RTO
Interview Tips
- • Given a scenario, state RPO/RTO and the exact tool covering each layer
- • Explain why manifest recovery (GitOps) is not data recovery — a common trap in interviews
Cheat Sheet
RPO = data-loss window → backup cadence; RTO = restore time → automation. Layers: etcdctl snapshot, Velero (objects+PVs), IaC, multi-cluster. Drill restores; measure RTO.
Continue Learning
Before You Start
- No prerequisites listed.