disaggregated prefill + decode — NVIDIA Dynamo / llm-d
router
cache-aware
+ SLA planner
prefill pool — compute-bound
H100 / B200
· matmul-heavy forward
· FLOPs-limited
· scale on queue depth
~2000 TFLOPS FP8 useful
decode pool — memory-bound
H200 or aggressive quant
· one token per iter, all weights
· HBM-bandwidth-limited
· scale on KV utilization
~3 TB/s HBM3 ceiling
prompt
NIXL
KV transfer
RDMA or TCP
NVIDIA Dynamo
· sits above vLLM / SGLang / TRT-LLM
· Planner Profiler + SLA Planner auto-configs
· Rust core, Python extensibility
· 30x on DeepSeek-R1; 50x MoE on GB300 NVL72
llm-d (Red Hat + AWS)
· Kubernetes-native Services per role
· packDomain: rack for KV locality
· per-role HPA (queue / KV util)
· 0.5: hierarchical KV, LoRA routing, UCCL
when it pays off
prompts > 512 tokens AND outputs > 200 tokens
MoE serving (DeepSeek-V3, future GPT-5 variants) — double win on expert routing
real case: $2M → $1.2M/yr on same workload, same SLA, no new hardware
short prompts: transfer tax dominates, do not disaggregate