One radix tree on one L4
The three SGLang processes run on the host CPU: the TokenizerManager, the Scheduler, and the DetokenizerManager, joined by ZMQ IPC. Below them is one NVIDIA L4 card with its SM array of 58 SMs and its 24 GB of GDDR6 drawn as one memory slab. In that memory the radix tree holds KV prefixes. The system prompt is one shared node of 2,998 tokens, and each repeated request attaches below it. A request with the id at the end attaches there too, with 2,986 of 3,033 tokens cached. A request with the id at the start shares only 6 tokens and starts its own branch. On our L4 the first request waited 2,143.9 ms for its first token, and the repeats waited 82.5 ms.
HOST CPU · THREE SGLANG 0.5.22 PROCESSES
Qwen3-8B
TokenizerManager
text to token ids
Scheduler
match_prefix, batches
DetokenizerManager
token ids to text
ZMQ IPC
x86_64 host
NVIDIA L4 · GPU MEMORY
SM array
58 SMs, Ada
24 GB GDDR6, six packages
root
system prompt node
2,998 tokens, shared
id at the start: 6 of 3,032 cached, own branch
match_prefix
repeat request, round 3
1 new token, 2,998 of
2,999 cached
id at the end: 2,986 of 3,033
ZMQ IPC
TIME TO FIRST TOKEN ON OUR L4, SAME 2,999-TOKEN PROMPT
first request 2,143.9 ms
repeats 82.5 ms, median of 4
first request: 3 cached tokens, the system prompt was not yet in the tree