Cluster Upgrades

Version Skew Rules

Kubernetes releases three times a year, and the compatibility rules constrain your plan:

• kubelet must not be newer than kube-apiserver and may be up to three minor versions older.
• kubectl supports one minor version either direction against the apiserver.
• You cannot skip minors: 1.28 to 1.30 passes through 1.29.

That yields the canonical order: backup etcd → apiserver → scheduler and controller-manager → kube-proxy and CoreDNS → kubelet, node by node. Read release notes per hop; deprecated APIs such as batch/v1beta1 CronJob must be migrated before the apiserver drops them, or those objects stop serving.

Executing the Upgrade

kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data
# on the control-plane node:
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.30.4
sudo apt-mark unhold kubelet kubeadm
sudo apt-get install kubelet=1.30.4-1.1 kubeadm=1.30.4-1.1
sudo systemctl restart kubelet
kubectl uncordon node-2

Repeat drain → upgrade kubelet → uncordon per worker while checking kubectl get nodes for the new version and kubectl get pods -A for anything CrashLooping. Managed providers compress this into a control-plane action plus a node-group roll.

A complete etcd snapshot taken first is your rollback path — control planes do not support downgrade.

Pitfalls and Rollback

Pitfall 1 — draining the last schedulable capacity. Roll one node at a time, respect PodDisruptionBudgets, and expect prompts: unmanaged DaemonSets are skipped with --ignore-daemonsets, and pods with local storage need --delete-emptydir-data or eviction stalls.

Pitfall 2 — kubelet before apiserver. That inverts the skew rule and can destabilize the node. apiserver first, kubelet last.

Interview-ready: recite the skew rules, the upgrade order, and your verification commands. For failure: pause the roll, inspect kubectl describe node and kubelet logs, restore etcd from the pre-upgrade snapshot if state is corrupt — and explain why downgrades are unsupported once storage versions migrate forward.