Requests served and requests served within the target, against offered load
Poisson arrivals at 0.5 to 16 requests per second on a GB10 with Qwen3-30B-A3B. Dashed lines show requests completed per second, solid lines show the requests that met a first token under 1 second and a mean time per token under 100 milliseconds. FP8 serves 1.03 requests per second within the target at 1 offered and almost none from 4. BF16 meets the target for almost no request at any load.
0
3
6
9
0.5
1
2
4
8
16
offered request rate, requests per second (Poisson)
requests per second
8.26 served
3.88 served
FP8 at 1 offered: 1.03 within target
FP8
BF16
served (dashed)
within target (solid)
target: first token under 1000 ms and mean time per token under 100 ms, per request