RadixAttention — KV cache as a tree, scheduler as the hot-branch pinner
the radix tree
SYSTEM (2000 tok)
TOOLS (300 tok)
DOC_A (500)
DOC_B (500)
DOC_C (500)
Q_1
Q_2
Q_3
Q_4
Q_5
Q_6
new request: SYSTEM + TOOLS + DOC_B + Q_7
walk the tree : SYSTEM reuse, TOOLS reuse, DOC_B reuse
allocate blocks only for Q_7 (60 tokens, 4 blocks)
prefill cost : 60 tokens instead of 2860
on prefix-heavy RAG : up to 6.4x SGLang over vLLM
the eviction policy
branch-level LRU : evict whole leaves
keeps cache shape matched to tree shape
cache-aware scheduling
FCFS is wrong for prefix-heavy traffic
serves requests in arrival order
evicts hot branches before they are reused
depth-first dispatch
prefer requests rooted at the running branch
keep the hot branch resident; stream siblings
approximates radix depth-first traversal
numbers (2026)
Llama 3.1 8B H100 ShareGPT 1K :
SGLang ~16,200 tok/s vs vLLM ~12,500 (+29%)
prefix-heavy RAG : up to 6.4x
voice cloning : 86.4% prefix-cache hit rate
production : 50-99% depending on template discipline
the gotcha — prompt ordering
[system, tools, context] ≠ [system, context, tools]
tree sees two distinct paths
6.4x disappears, back to vLLM throughput
engineer's lever : fix the template
immutable first (system, tools, schemas)
user input last; real case 7% to 74% in one change