The same open loop on a 24 GB L4
Poisson arrivals at 0.5 to 16 requests per second on one L4 with Qwen3-8B. Dashed lines show requests completed per second, solid lines show the requests that met a first token under 1 second and a mean time per token under 100 milliseconds. FP8 serves 3.18 requests per second within the target at 4 offered. BF16 peaks at 0.96 within the target at 1 offered, and completes fewer requests as the offered rate rises past 4.
0
2
4
6
0.5
1
2
4
8
16
offered request rate, requests per second (Poisson)
requests per second
5.61 served
1.62 served
FP8 at 4 offered: 3.18 within target
FP8
BF16
served (dashed)
within target (solid)
target: first token under 1000 ms and mean time per token under 100 ms, per request