FP8 moves the whole latency and throughput curve, not one point
Each dot is one concurrency from 1 to 64 on a GB10 with Qwen3-30B-A3B, plotted as output throughput against median time per output token. At every concurrency the FP8 dot sits to the right and lower: more tokens per second for less time per token.
0
50
100
150
200
250
300
0
100
200
300
400
500
output tokens per second
median TPOT, ms
c1
c64
c1
c64
BF16 weights, BF16 KV
FP8 weights, FP8 KV