A prefix hit reuses whole 16-token blocks up to the first difference
Two prompts share a system prompt. Each full 16-token block gets a hash that includes the hash of the block before it. The second prompt finds the first three block hashes in the map and reuses those blocks. Its fourth block differs, so the chain stops and the rest is computed.
REQUEST A, ALREADY SERVED
REQUEST B, ARRIVES LATER
block 1
h1
cache hit
h1
block 2
h2
cache hit
h2
block 3
h3
cache hit
h3
block 4
h4a
compute
h4b
block 5
h5a
compute
h5b
differs here, chain stops
Each hash covers its 16 tokens and the hash before it, so h3 names the first 48 tokens.
A shared part of 20 tokens gives one cached block, and 4 tokens are always computed.
Put the shared part first, in the same order, to keep the chain long.