Cold start mitigation — five layers on a 70B model
raw cold start — ~328 s total (5.5 min)
node provision
50 s
image pull
180 s
weights to HBM
75 s
engine init
20 s
1st fwd
3 s
L1 — pre-seeded image
Bottlerocket dual volume
snapshot data volume with image
EC2NodeClass references snapshot
eliminates 180s image pull
L2 — Model Streamer
Run:ai / native vLLM 2026
stream weights layer-by-layer
overlap I/O with compute setup
~2x weights-load speedup
L3 — GPU snapshot
Modal checkpoints
serialize post-load HBM state
up to 10x faster restart
per-GPU-topology
L4 — warm pool
min_workers ≥ 1
eliminates cold path entirely
cost: 24x7 GPU rental
mandatory at P99 TTFT < 60s
L5 — tiered loading
ServerlessLLM pattern
NVMe → DRAM → HBM pipeline
10-200x latency reduction
early production adoption
bonus — live migration
move tokens, not cache
send input (KB) to warm dest
recompute KV on destination
useful for disaggregated
stack result — ~15 s with L1+L2+L3 or full warm via L4
Modal published: 2-4 s · Baseten default: 5-10 s, sub-s with pre-warming
the right answer depends on SLA shape and warm-pool budget
measure before optimizing — never add layers without data