One benchmark, two cache settings
Three pairs of bars for the shared-prefix benchmark on the capture Mac, radix cache on against --disable-radix-cache. Duration is 48.81 s against 129.68 s, median time to first token is 1,358 ms against 7,222 ms, and output throughput is 83.91 against 31.59 tokens per second.
radix cache on
--disable-radix-cache
BENCHMARK DURATION
48.81 s
129.68 s
MEDIAN TIME TO FIRST TOKEN
1,358 ms
7,222 ms
OUTPUT TOKENS PER SECOND
83.91 tok/s
31.59 tok/s
64 prompts · 4 groups of 16 · 2,048-token shared system prompt · max concurrency 8