Blackwell + TRT-LLM stack — where the 7x cost gap comes from
precision regime on Blackwell
weights : NVFP4 (4-bit microscaling)
half the bytes per decoded token vs FP8
day-0 FP4 weights shipped by model providers
KV cache : FP8 (stays critical)
FP4 KV collapses attention tails
FP8 exponent bits preserve dynamic range
activations : NVFP4
small-range within a layer; FP4 holds up
MXFP8 variant available on Tensor Cores
attention accumulator : FP32
softmax stability requires higher precision
no negotiation on this tier
the quality caveat
NVFP4 weights on reasoning workloads :
math, chain-of-thought, long-context code :
visible quality drop without per-block calib
recovery : FP8 weights + FP4 activations
or stay on H200 with FP8 throughout
always validate eval quality pre-migration
stack layers that stack the gap
1 / hardware bandwidth
H100 3.35 TB/s vs B200 8.0 TB/s : ~2.4x
decode is memory-bound; this is the floor
2 / NVFP4 weights
halve bytes per token : ~2.0x
NVFP4 is the only weight precision that matters now
3 / MTP draft
multi-token prediction head : ~1.8x on accepted tokens
integrated in the TRT-LLM build
4 / disaggregated prefill/decode
Dynamo orchestrates separate GPU pools : ~1.6-2.5x
NVLink 5 for KV cache transfer
5 / NVLink 5 all-to-all
MoE expert comm latency cut 3x vs Hopper
MoE-heavy models get extra multiplier
headline numbers (2026)
HGX B200 : $0.02/M tok GPT-OSS-120B
GB200 NVL72 + Dynamo : $0.012/M tok
H100 + vLLM : $0.09/M tok