The same choice at a fixed arrival rate
Median time to first token and P99 time per output token against a Poisson arrival rate from 0.5 to 16 requests per second for SGLang 0.5.22 and vLLM 0.31.0 with Qwen3-8B on one NVIDIA L4, with KV pools of 24,094 and 24,096 tokens, 512 input and 128 output tokens, and three seeds per point, on a log scale. Up to 2 requests per second all four lines stay under 1,000 ms. From 4 requests per second the first-token lines rise to many seconds, vLLM lower, and the vLLM P99 time per token rises above the SGLang line, 187 against 127 ms at 4.
50
100
300
1k
10k
100k
0.5
1
2
4
8
16
offered request rate, requests per second (Poisson)
ms (log scale)
SGLang median TTFT
vLLM median TTFT
SGLang P99 TPOT
vLLM P99 TPOT