One block pool on one L4
The vLLM 0.31.0 API server and EngineCore run on the host CPU and talk over ZMQ. The Scheduler inside EngineCore builds one step under a token budget of 2,048 tokens: requests A and B decode one token each, and request C adds a prefill chunk. The block table of the KVCacheManager maps each request to blocks that are not all next to each other. Below is one NVIDIA L4 with its SM array of 58 SMs and 24 GB of GDDR6, where the KV cache is a grid of 16-token blocks: 1,506 blocks hold 24,096 tokens, and the grid shows 40 of them: the blocks of A and B, some of the blocks of C, and free blocks.
HOST CPU · VLLM 0.31.0 PROCESSES
API server
HTTP, tokenizer
EngineCore
Scheduler
KVCacheManager
block table
A · 2, 13, 31
B · 7, 8, 26, 38
C · 17, 18, 27, 28, 35, more
ZMQ
allocate_slots
x86_64 host
ONE SCHEDULER STEP
budget 2,048 tokens
A decode, 1 token
B decode, 1 token
C prefill chunk, 2,046 tokens
The decode slivers are drawn wider. Decodes go first, then a prompt chunk fills the rest.
NVIDIA L4 · GPU MEMORY
SM array
58 SMs, Ada
KV cache · 16-token blocks
1,506 blocks, 24,096 tokens
24 GB GDDR6, six packages
in use
free · 40 of 1,506 shown
With 8 users the log read: GPU KV cache usage 77.2%, prefix cache hit rate 87.2%.
Block ids show the mapping only. Pool, block size, and budget are from our L4 runs.