When Nodes Are Added or Removed
When Nodes Are Added or Removed
The Cluster Autoscaler runs in-cluster and acts as a client of your cloud API. Scale-up happens when pods sit Pending with Insufficient cpu/memory and some node group can satisfy them below its maximum size. Scale-down requires a node to be underutilized — total requests below --scale-down-utilization-threshold (default 0.5) — and unneeded for --scale-down-unneeded-time (default 10m).
Before evicting, it respects PodDisruptionBudgets and pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false". It will not delete a node it cannot bring back (group at max) or one hosting non-mirror pods it cannot reschedule.
Deploying and Tuning
Deploying and Tuning
Install as a Deployment in kube-system with its RBAC manifests, then point it at your cloud:
# AWS EKS: node groups discovered via ASG tags
--cloud-provider=aws
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled, k8s.io/cluster-autoscaler/my-cluster
Key flags: --expander=least-waste (pick the node group that leaves the least idle capacity), --balance-similar-node-groups=true, and --scale-down-delay-after-add. Disable scale-down on a critical node with the annotation cluster-autoscaler.kubernetes.io/scale-down-disabled: "true". Watch its logs to see which condition (pending pod vs unneeded node) it is currently evaluating.
Pitfalls and Interview Angles
Pitfalls and Interview Angles
Pending but never scaled. The pod may need a zone or instance type no node group offers, wait on a PVC that cannot bind, or violate a taint no group matches — autoscaling only fixes capacity shortfalls the groups can actually fill.
Scale-down blocked. A PDB with minAvailable matching current replicas stops every eviction, so the node lingers. So do DaemonSet-only nodes computed above the threshold.
Wrong tool. Cluster Autoscaler reacts to unschedulable pods; it does not act on CPU spikes. For request-driven bursting use KARPENTER or HPA-plus-pool logic.
Interview angle: state the two trigger conditions (pending pod, unneeded node), the flags that tune them, and how PDBs gate eviction.
Quiz
1. What immediately triggers a scale-up attempt?
2. A node stays for hours below 50% utilization. Which default is most likely blocking removal?
3. What does --expander=least-waste do?
Flashcards
Question
Scale-up trigger
Click to reveal answer
Answer
Pods Pending with insufficient resources plus a node group under its max that can fit them.
Question
Scale-down trigger
Click to reveal answer
Answer
Node requests/allocatable below --scale-down-utilization-threshold (0.5) for --scale-down-unneeded-time (10m), subject to PDBs.
Question
safe-to-evict annotation
Click to reveal answer
Answer
cluster-autoscaler.kubernetes.io/safe-to-evict: "false" prevents the autoscaler from evicting that pod during scale-down.
Question
Expander strategies
Click to reveal answer
Answer
random, price, most-pods, least-waste — scoring rules for picking a node group at scale-up.
Revision Notes
Key Takeaways
- 1. Scale-up = unschedulable pod + capable node group; scale-down = sustained low utilization + evictable pods
- 2. PDBs and safe-to-evict annotations gate removal as much as utilization numbers do
- 3. The autoscaler fixes capacity, not traffic spikes — pair with HPA/Karpenter for demand response
Interview Tips
- • Give both trigger conditions and the two flags that tune them before discussing cloud specifics
- • Explain why a PDB can make scale-down a no-op, and how you would verify it from autoscaler logs
Cheat Sheet
CA: scale-up on Pending+insufficient if node group < max; scale-down after 10m below 0.5 utilization, respects PDBs. Flags: --expander=least-waste, --balance-similar-node-groups. Annotate to disable.