The longer the shared prompt, the more the tree saves
Median time to first token against the length of a shared system prompt on one NVIDIA L4 with Qwen3-8B, three seeds per point, on a log scale. With the radix cache the median stays between 349 and 479 ms. Without it the median grows from 379 ms with no shared prompt to 14,706 ms with 4,096 shared tokens.
100
300
1k
3k
10k
30k
0
1k
2k
3.1k
4.1k
shared system prompt, tokens
median TTFT, ms (log scale)
479 ms
14,706 ms
radix cache on
--disable-radix-cache