One prefix through the three HiCache tiers
Three tiers from left to right: L1 is the GPU KV pool, L2 is pinned host memory private to the instance, and L3 is a storage backend such as file, mooncake, hf3fs, or nixl. Under write_through, a node is copied to L2 on its first hit, and write_through_selective waits for the second hit. Eviction from L1 leaves the host copy. A later match loads the host part back into fresh GPU slots one layer ahead of compute. A storage prefetch is skipped when the L3 hit is under 256 tokens.
L1 · GPU pool
device memory
requests and cache
L2 · host
pinned memory
one instance only
L3 · storage
shared by servers
file, mooncake
WHEN A PREFIX GETS A HOST COPY
write_through (default)
on its first hit · threshold 1
write_through_selective
on its second hit · threshold 2
write_back
only when L1 evicts it
STORAGE PREFETCH AT V0.5.22
skipped under 256 matching tokens · stops after 1.0 s + 0.25 s per 1,024 tokens
copy
back up
fetch
load