HiCache held the prompts that the GPU pool evicted
Median time to first token against the number of distinct 2,048-token system prompts on one NVIDIA L4 with Qwen3-8B, three seeds per point, on a log scale. The GPU pool holds 24,094 tokens. At 4 prompts both servers take about 360 ms. At 16 and 32 prompts the default takes 3,259 and 5,449 ms and HiCache 553 and 817 ms. At 64 prompts, more than the GPU and host pools hold together, the default takes 6,821 ms and HiCache 3,610 ms.
100
300
1k
3k
10k
4
8
16
32
64
distinct 2,048-token system prompts
median TTFT, ms (log scale)
GPU pool only
HiCache, host pool of 96,377 tokens