Scale up is the easy half of the Cluster Autoscaler: a pod sits Pending for lack of capacity, the controller requests a node, the node shows up. Scale down is the half that fails quietly: a node sitting underused for hours, never reclaimed, nothing obvious in the logs unless you know where to look.

What triggers a scale up

The controller watches for Pending pods the scheduler can’t place given their requests, affinities, and taints. It then simulates adding a node from each available node group’s template, and provisions whichever group would let the pending pods be scheduled. Without correctly set requests on the pods (the same silent prerequisite as HPA), that simulation has no reliable basis.

What blocks a scale down

A node is only a removal candidate once its utilization stays under a threshold for a sustained duration (scale-down-unneeded-time, 10 minutes by default) and every pod on it can be rescheduled elsewhere without violating a constraint. That second condition is where it silently stalls:

  • An overly strict PodDisruptionBudget (minAvailable equal to the replica count, for instance) blocks any eviction, and therefore any drain of the node hosting them.
  • A pod managed by no controller (created directly, with no Deployment or ReplicaSet behind it) has no guarantee of being recreated elsewhere; the Cluster Autoscaler refuses to evict it by default.
  • A local volume or an emptyDir holding data considered important makes the pod non-movable without loss, unless explicitly annotated to accept eviction.
  • Restrictive anti-affinity rules can leave a pod with nowhere else to schedule, which blocks its removal indefinitely.
# See why a specific node isn't being removed: the Cluster Autoscaler's
# detailed status lists the reasons, node by node.
kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

The annotation everyone forgets

A pod running standalone, with no controller, or a homegrown DaemonSet-like workload, can explicitly unblock its own eviction:

metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"

Conversely, a pod whose eviction would be dangerous (an in-progress migration job, say) can explicitly refuse it with the same annotation set to "false": safer than relying on the absence of a controller to produce the same effect by accident.

Multiple node groups: the expander

With several eligible node groups (different instance types, different zones), the Cluster Autoscaler has to pick which one to grow. The default strategy (random) reflects no business priority; the priority expander lets you define an explicit order via regex on group names, useful for preferring a cheaper node pool before falling back to a more expensive one.

What this doesn’t fix

Scale up and scale down both assume node groups are configured with realistic min/max bounds and instance types that can actually fit the workloads waiting on them. A node group capped at a max too low for real demand produces Pending pods the Cluster Autoscaler can see and explain but cannot resolve: the fix is in the node group configuration, not the controller.

Takeaway

Cluster Autoscaler scale up depends on the same honest requests as HPA. Scale down depends on a much more fragile condition: that every pod on a node can be rescheduled elsewhere without violating a PDB, an affinity constraint, or a controller guarantee. A node that refuses to leave is almost never a controller bug: it’s a constraint set somewhere else doing exactly its job. Getting this right is part of what’s decided during a Kubernetes migration.