Reconciliation as a Control Loop
Reconciliation as a Control Loop
An operator pairs a CRD with a controller that encodes operational knowledge. The loop never waits for a perfect event stream: it receives a request (create/update/delete/periodic resync), fetches the object, compares spec (desired) to the cluster (actual), acts, and writes status.
It is level-triggered, not edge-triggered: "make it so" at any moment, tolerating missed events. Every action must be idempotent — Create failing with AlreadyExists is success, not an error. Errors requeue with exponential backoff; success schedules a periodic resync so drift outside the event stream still gets noticed.
Status, Owners, and Finalizers
Status, Owners, and Finalizers
Report progress through a status subresource with metav1.Condition entries:
status:
observedGeneration: 4
conditions:
- type: Ready
status: "True"
reason: BackupComplete
lastTransitionTime: "2026-09-01T10:00:00Z"
observedGeneration ties status to the spec version clients submitted — stale conditions are visible immediately.
OwnerReferences put the controller's object in charge of dependents: delete the custom resource and Kubernetes garbage-collects the Services/ConfigMaps it owns (with blockOwnerDeletion protecting the owner during cascades). Finalizers run the opposite direction: they hold deletion of the custom resource itself until the controller finishes cleanup (drop connections, snapshot data), then remove the finalizer to let deletion proceed.
Pitfalls and Interview Angles
Pitfalls and Interview Angles
Non-idempotent reconciles. Creating resources without treating AlreadyExists as convergence turns every resync into an error loop.
Stuck finalizers. If the controller is gone when a delete arrives, objects hang in Terminating forever — document the manual kubectl patch escape hatch.
Status thrash. Updating status when nothing changed causes infinite reconciles (the update triggers another event); only write on transitions and bump observedGeneration deliberately.
When not to build one. If the workflow is "install these manifests once," use Helm; operators earn their complexity through day-2 automation — failover, backups, upgrades.
Interview angle: walk reconciliation of a Database CR: create StatefulSet, wait Ready, publish connection info in status, and clean up on delete via finalizer.
Quiz
1. What makes reconciliation level-triggered rather than edge-triggered?
2. What should a reconcile do when Create returns AlreadyExists?
3. Why use a finalizer on a custom resource?
Flashcards
Question
Level-triggered reconciliation
Click to reveal answer
Answer
Converge from any observed state to desired spec; robust to missed or duplicate events.
Question
observedGeneration
Click to reveal answer
Answer
Status field recording which spec generation the controller has processed — detects stale status.
Question
ownerReferences
Click to reveal answer
Answer
Links dependents to their owner so Kubernetes garbage-collects children when the CR is deleted.
Question
Finalizer vs ownerReference
Click to reveal answer
Answer
Finalizer delays the owner's own deletion for cleanup; ownerReference causes children to be deleted with the owner.
Revision Notes
Key Takeaways
- 1. Operators are idempotent, level-triggered loops driving spec to reality and reporting in status
- 2. Conditions + observedGeneration make progress observable to humans and other controllers
- 3. OwnerReferences handle children; finalizers handle the CR's own deletion — both have failure modes
Interview Tips
- • Walk a full reconcile of a domain CR, naming each Kubernetes primitive you rely on
- • Discuss what happens when your controller dies mid-deletion — finalizer stuck, and how you recover
Cheat Sheet
Operator = CRD + reconcile loop: fetch → diff spec/status → act idempotently → write conditions (observedGeneration). ownerReferences GC children; finalizers gate CR deletion. Requeue with backoff.