Skip to content

node-autoscaling.md

Latest commit

 

History

History
251 lines (205 loc) · 11.4 KB

node-autoscaling.md

File metadata and controls

251 lines (205 loc) · 11.4 KB
Raw

Node Autoscaling

Automatic horizontal and vertical scaling of Hetzner worker nodes via the Kubernetes Cluster Autoscaler with the Hetzner Cloud provider.

KSail natively installs and manages the Cluster Autoscaler when spec.cluster.autoscaler.node.enabled: true is set in the ksail config.

Architecture

KSail (static baseline)
├── 3 control planes (cx33, 4 vCPU / 8 GB, never autoscaled)
└── 3 static workers (cx33, 4 vCPU / 8 GB, guaranteed minimum, Longhorn storage nodes)

Cluster Autoscaler (dynamic workers, managed by KSail)
├── Pool: autoscale-cx43 → 0-4 × CX43 (8 vCPU, 16 GB)
├── Pool: autoscale-cx53 → 0-4 × CX53 (16 vCPU, 32 GB)
├── maxNodesTotal: 10 (total cluster nodes incl. baseline: 6 static + 4 autoscaler)
└── Expander: [LeastNodes, LeastWaste] (priority chain)
  • Horizontal scaling — autoscaler adds workers when pods are Pending due to insufficient resources, and removes underutilized workers after a configurable cooldown.
  • Vertical scaling — multiple node pools with different server types. The expander is an ordered priority chain, [LeastNodes, LeastWaste] (upstream --expander=least-nodes,least-waste). LeastNodes runs first and keeps the pools that satisfy the pending pods with the fewest total new nodes — preferring the largest adequate type so a burst consolidates onto fewer, bigger servers; LeastWaste then breaks any tie by least idle CPU/memory. (Price is not supported on Hetzner: the cluster-autoscaler hcloud provider implements no pricing API, so KSail rejects it and the autoscaler crashes on startup.) See cluster-autoscaler FAQ.
  • KSail integration — KSail installs the Cluster Autoscaler Helm chart, generates the worker config secret (cluster-autoscaler-config), and manages the Talos snapshot lifecycle. Node pool configuration lives in ksail.prod.yaml, not in Flux manifests.
  • Storage architecture — autoscaler nodes are compute-only (no Hetzner volume, no Longhorn storage). Static KSail workers have dedicated Hetzner volumes and serve as Longhorn storage nodes. Pods on autoscaler nodes access Longhorn PVCs via the CSI driver (network). The Hetzner Cluster Autoscaler does not support volume attachment.

How new nodes join

  1. Cluster Autoscaler detects Pending pods with unmet resource requests.
  2. It calls the Hetzner API to create a new server using:
    • HCLOUD_IMAGE — a Talos snapshot managed by KSail.
    • HCLOUD_CLOUD_INIT — base64-encoded Talos worker machine config generated by KSail.
  3. The server boots Talos, applies the machine config, and joins the cluster.
  4. Once the node is Ready, pending pods are scheduled.

Configuration

All autoscaler configuration lives in ksail.prod.yaml under spec.cluster.autoscaler.node:

spec:
  cluster:
    autoscaler:
      node:
        enabled: true
        expander: [LeastNodes, LeastWaste]
        maxNodesTotal: 10
        scaleDownUnneededTime: "10m"
        pools:
          - name: autoscale-cx43
            serverType: cx43
            location: fsn1
            min: 0
            max: 4
          - name: autoscale-cx53
            serverType: cx53
            location: fsn1
            min: 0
            max: 4
Field Default Description
enabled false Enable/disable node autoscaling
expander LeastWaste Node selection strategy: LeastWaste, LeastNodes, Random (Price is unsupported on Hetzner — no pricing API). Accepts a single value or an ordered priority chain, e.g. [LeastNodes, LeastWaste] (requires KSail expander-list support; scalar-only up to v7.57.0)
maxNodesTotal 0 (unlimited) Hard ceiling on total cluster nodes, including the static baseline (see ksail#5017)
scaleDownUnneededTime 10m Time before an underutilized node is eligible for removal
pools[].name — DNS-1123 pool identifier
pools[].serverType — Hetzner server type (e.g., cx23, cx33)
pools[].location — Hetzner datacenter (e.g., fsn1)
pools[].min — Minimum nodes in pool
pools[].max — Maximum nodes in pool

Cost guardrails

  • Hard max per pool — pools[].max caps each pool independently. Set to 4 so any single CX type can serve a full burst (e.g. 4× cx23) instead of forcing larger types; the maxNodesTotal total ceiling still caps the autoscaler at 4 (10 − 6 baseline), so this changes only the type distribution, never the node count.
  • Total node ceiling — maxNodesTotal caps the total cluster node count, including the static baseline. Set to 10 (6 static + up to 4 autoscaler). It is passed straight to cluster-autoscaler's --max-nodes-total, so the runtime already enforces this total.
  • serverLimit (spec.provider.hetzner.serverLimit) — the Hetzner account hard cap (10). Under the in-progress KSail change (ksail#5017) the autoscaler validation becomes maxNodesTotal ≤ serverLimit (10 ≤ 10); until it ships, the old validation (CP + workers + min(maxNodesTotal, Σ pool.max)) rejects this config, so the KSail change must land first.
  • Expander — [LeastNodes, LeastWaste] (current) is an ordered priority chain. LeastNodes runs first and keeps the pools that scale up with the fewest total new nodes (preferring the largest adequate type to keep the node count down); LeastWaste then breaks any remaining tie by least idle CPU/memory. The list form needs KSail expander-list support — releases up to v7.57.0 are scalar-only and reject a list, so the pinned KSAIL_VERSION must be bumped to a release that ships it first. (Price is unsupported on Hetzner.)
  • Scale-down — underutilized nodes are removed after 10 minutes.

Adding more pools

Add a new entry to the pools list in ksail.prod.yaml and run ksail cluster update. KSail updates the Helm release automatically.

Pod rebalancing & consolidation (descheduler)

The Cluster Autoscaler only ever adds or removes nodes — it never moves a running pod. Over time placement degrades (replica clumps after a node failure, pods left on nodes that now violate their affinity, half-empty autoscale-cx* nodes the autoscaler can't drain because one pod is stuck there). On clouds with Karpenter that gap is closed by its consolidation feature, but Karpenter has no Hetzner provider — so the platform pairs the autoscaler with the Kubernetes SIG Descheduler instead: the descheduler evicts misplaced pods (the scheduler re-places them), and the autoscaler reclaims any node that empties out (scaleDownUnneededTime: 10m).

It runs as a HelmRelease in kube-system (k8s/providers/hetzner/infrastructure/controllers/descheduler/), as a single-replica Deployment on a 30-minute descheduling loop. Enabled strategies:

  • Placement repair — RemovePodsViolatingNodeAffinity (required affinity), RemovePodsViolatingNodeTaints, RemovePodsViolatingInterPodAntiAffinity, RemovePodsViolatingTopologySpreadConstraint (hard constraints only), and RemoveDuplicates (re-spread replica clumps after node churn).
  • Consolidation — HighNodeUtilization drains nearly-empty nodes (< 15% requested CPU and memory) toward the rest of the cluster so the autoscaler can delete them. The threshold is deliberately low: with the default scheduler scoring, evicted pods gravitate back toward empty-ish nodes (including the over-provisioning warm spare), so aggressive packing needs the MostAllocated scoring strategy first — tracked in #2471.

Guardrails (all in the HelmRelease values):

  • Evictions go through the eviction API, so PodDisruptionBudgets are always honored.
  • PVC-backed pods are never evicted (PodsWithPVC protection): CNPG clusters, Longhorn-attached workloads, openbao, spire-server. Stateful moves stay deliberate runbook operations (see docs/dr/).
  • All default protections stay on: DaemonSet, mirror/static, system-critical, terminating, local-storage, and failed bare pods are untouchable.
  • nodeFit: true — a pod is only evicted if it provably fits on another node.
  • Blast-radius caps: at most 2 evictions per node and 2 per namespace per 30-minute cycle.

Evictions surface as Kubernetes Events (visible in Coroot); watch for pods bouncing between nodes every cycle — that's the signal a threshold is fighting the scheduler and should be tuned down.

Troubleshooting

Autoscaler not scaling up

# Check autoscaler logs
kubectl -n kube-system logs -l app.kubernetes.io/name=cluster-autoscaler --tail=100

# Check for unschedulable pods
kubectl get pods -A --field-selector=status.phase=Pending

# Check autoscaler status ConfigMap
kubectl -n kube-system get cm cluster-autoscaler-status -o yaml

Autoscaler nodes not joining

# Check if the Hetzner server was created
hcloud server list --selector cluster.autoscaler.nodeGroupLabel

# Check Talos bootstrap status (if server IP is known)
talosctl -n <node-ip> health

# Verify the machine config is valid
talosctl validate --config worker.yaml --mode cloud

Cluster rebuild

After a full cluster rebuild (ksail cluster delete + create):

  1. KSail regenerates the worker config secret automatically.
  2. KSail manages the Talos snapshot lifecycle — no manual snapshot creation is needed.

Maintenance

Talos version upgrades

When bumping the Talos (or Kubernetes) version in ksail.prod.yaml, the deploy's ksail cluster update brings both node classes onto the new baseline:

  1. KSail's snapshot lifecycle manager creates or updates the Talos snapshot automatically during cluster update, and the worker machine config is regenerated to match — so new autoscaler nodes boot the new version.
  2. The static control planes and workers are upgraded in place (rolling).
  3. Existing autoscaler nodes are recycled automatically so they follow the new baseline instead of drifting on the old version: after the refreshed cluster-autoscaler is ready, KSail cordons and drains each autoscaler node one at a time (via the Kubernetes eviction API, honoring PodDisruptionBudgets) and deletes its Hetzner server; the autoscaler then re-provisions any still-needed capacity from the new snapshot on demand. This runs only when the version actually changes — a no-op cluster update leaves autoscaler nodes untouched. See the KSail Autoscaler Node Upgrades docs.

A strict PodDisruptionBudget on a workload running on autoscaler nodes can slow or block the drain (the update fails rather than force-evicting), so keep PDBs realistic for compute-only/burstable workloads.

Hetzner server type changes

Hetzner periodically renames or retires server types. Check the Hetzner Cloud changelog and update the pools[].serverType values in ksail.prod.yaml.