On the GB10 the worst token gap of vLLM doubled from 32 users
The 99th percentile inter-token latency against users at the same time on a DGX Spark with Qwen3-30B-A3B, three seeds per point. The engines match up to 8 users. At 32 users vLLM shows 478 ms and SGLang 220 ms, and at 64 users 528 ms and 271 ms.
0
200
400
600
1
2
4
8
16
32
64
users at the same time
P99 inter-token latency, ms
478
220
vLLM 0.31.0
SGLang 0.5.22