Two groups of the shared-prefix benchmark in one tree
A radix tree with a root that has lock count 1 forever. Under it, the group 1 system prompt node holds 2,048 tokens and has lock count 2, because two running requests use it. It splits into two leaves of 128 question tokens, each with lock count 1. A second branch holds the group 2 system prompt with lock count 0, so it is cached and evictable. Eviction removes leaves with lock count 0 first, from the tip toward the root.
root
lock 1, forever
group 1 system prompt
2,048 tokens · lock 2
question A
128 tokens · lock 1
question B
128 tokens · lock 1
match_prefix splits a node where
two token runs stop agreeing.
group 2 system prompt
2,048 tokens · lock 0 · evictable
A lock count above 0 protects the whole path to the root. evict frees leaves with lock 0
first, oldest access time first, and moves up the branch.