FP8 weights at 32 users with the clock locked at 900 MHz
Qwen3-8B in BF16 against Qwen3-8B-FP8 on an NVIDIA L4 at 32 users, SM clock locked at 900 MHz, KV pool fixed at 24,096 tokens, two seeds. Output was 208.8 tokens per second for BF16 and 284.7 for FP8. The median time per token was 119.9 ms for BF16 and 83.8 ms for FP8.
output tok/s, higher is better
0
200
400
208.8
BF16
284.7
FP8
median ms per token, lower is better
0
50
100
150
119.9
BF16
83.8
FP8
Both runs at 32 users with the same KV pool. Median of two seeds.