Cost Optimization

Where the Money Goes

Node capacity dominates the bill, so the first ratios matter: requests vs allocatable (are you paying for reservations nobody uses?) and usage vs requests (is reserved capacity actually consumed?). Typical waste appears as oversized requests that block bin-packing, idle non-prod clusters running 24/7, and orphaned volumes/LBs/IPs from deleted workloads.

Measure before acting:

sum(kube_pod_container_resource_requests{resource="cpu"})
sum(kube_pod_container_resource_requests) by (namespace)

OpenCost or Kubecost break the bill down by namespace/label — feed them allocation data, but the underlying levers are cluster-side: requests, autoscaling, and instance mix.

Rightsizing and Instance Mix

Run VPA in Off mode across workloads and apply its recommendations during maintenance windows — the cheapest CPU is the one you never request. For fault-tolerant stateless workloads use spot/preemptible capacity at deep discounts:

# node taint so only tolerant workloads land there
tolerations:
- key: spot
  operator: Exists
  effect: NoSchedule
nodeSelector: { spot: "true" }

Enable multiple instance types in the node group (or Karpenter NodePools) so reclaim capacity can be filled from a pool of shapes; handle interruption notices with graceful shutdown and PDBs. Baseline load fits reserved/committed-use instances; peaks ride spot and autoscaling — each tier matches a different cost curve.

Pitfalls and Interview Angles

Optimizing the wrong metric. Removing CPU limits to "save cost" does nothing to the bill (limits do not reserve) while changing throttling behavior — savings come from requests and node counts.

Committing to peaks. Buying committed-use for capacity that only exists during peaks over-pays; commits should cover measured baseline only.

Untagged resources. Without cost-allocation labels, showback debates become arguments; enforce label policies at admission.

Spot for the wrong workload. Databases and single-replica stateful sets on spot risk interruption outages; pair spot with disruption budgets and multiple replicas.

Interview angle: "our bill doubled" — walk the investigation: node count trend, request/usage ratio, namespace allocation, orphaned PVCs/LBs, then the fixes ranked by savings-to-risk.