On a 24 GB card the BF16 pool fills and throughput stops climbing
Output throughput against concurrency from 1 to 64 on one NVIDIA L4 with Qwen3-8B, three seeds per point with a band from the lowest to the highest run. The BF16 line climbs to 270 tokens per second at 32 and falls to 251 at 64. The FP8 line keeps climbing to 736 at 64.
0
200
400
600
800
1
2
4
8
16
32
64
max concurrency
output tokens per second
251
736
BF16, 24,094-token pool
FP8 weights and KV, 142,369-token pool