Same throughput, two different choices at a full pool
Median time to first token and P99 inter-token latency against concurrency from 1 to 64 for SGLang 0.5.22 and vLLM 0.31.0 with Qwen3-8B on one NVIDIA L4, with KV pools of 24,094 and 24,096 tokens and three seeds per point, on a log scale. Up to 16 the inter-token lines lie together and SGLang gives the first token sooner. At 32 and 64 vLLM gives the first token sooner, 1,813 against 2,871 ms and 10,953 against 18,356 ms, while its P99 inter-token latency rises to 619 and 1,363 ms against 102 and 114 ms for SGLang.
50
100
300
1k
3k
10k
30k
1
2
4
8
16
32
64
max concurrency
ms (log scale)
SGLang median TTFT
vLLM median TTFT
SGLang P99 ITL
vLLM P99 ITL