N-gram speculation cut the time per token to about a third for one user
Median time per output token on an NVIDIA L4 with Qwen3-8B, a small KV pool, two seeds, random prompts. At one user, 241.7 ms without speculation and 76.6 ms with n-gram. At four users, 272.6 ms and 101.3 ms. The GPU ran at 375 to 443 MHz in all four runs.
0
100
200
300
median ms per token
241.7
1 user, none
76.6
1 user, n-gram
272.6
4 users, none
101.3
4 users, n-gram
Lower is faster. Random prompts, acceptance counters not recorded.