GPU autoscaling — three layers, not one
layer 1 — node provisioning
Karpenter
provisions nodes on pending pods
45-60s (vs CA 90-120s)
consolidation trap
WhenEmptyOrUnderutilized
evicts running GPU jobs
loses minutes of capacity
safe setting
consolidationPolicy:
WhenEmpty
consolidateAfter: 1h
taint GPU nodes
picks instance type dynamically
layer 2 — gang scheduling
KAI Scheduler
secondary to kube-scheduler
annotate workloads to opt in
gang scheduling
all-or-nothing placement
prevents 7-of-8 idle trap
defer all if can't fit all
topology awareness
NVLink / IB / rack placement
TP-8 stays on one NVLink domain
hierarchical queues
multi-team priority + quota
preemption rules per queue
Ray + vLLM production-stack integrate
layer 3 — application signals
HPA trap
DCGM_FI_DEV_GPU_UTIL
duty-cycle, NOT load
100% = 10 or 100 requests
correct signals
queue depth
KV cache utilization
per-replica P99 TTFT
goodput (SLO-meeting RPS)
Dynamo Planner
scales on inference signals
rate-matches prefill:decode ratio
meets SLOs, not duty cycle
llm-d WVA
Workload Variant Autoscaler
disaggregated prefill + decode
per-role HPA
compose all three — each alone is insufficient on a real GPU cluster