A cap of 16 sequences against the default as users grow
Output tokens per second and median time to first token against users at the same time, on an NVIDIA L4 with Qwen3-8B, three seeds per point, 2026-10-21. The two configs match up to 16 users. At 32 users, the default gave 275.1 tokens per second with a first token in 1.8 s, and the cap of 16 gave 171.8 tokens per second with a first token in 12.3 s.
default
--max-num-seqs 16
0
100
200
300
1
4
16
64
users
output tok/s
0.1
1
10
100
1
4
16
64
users
median first token, s