Time per output token grows with the batch on a mixture-of-experts model
Median time per output token for Qwen3-30B-A3B on a GB10. BF16 goes from 32.5 milliseconds at concurrency 1 to 254 at 64, about eight times. FP8 goes from 18.4 to 132. Each extra request routes its tokens to other experts, so each decode step reads more expert weights.
0
50
100
150
200
250
300
1
2
4
8
16
32
64
max concurrency (benchmark client)
median TPOT, ms
254
132
BF16
FP8