vLLM scheduler — three compounding defaults
PagedAttention allocator
KV cache = blocks of 16 tokens
block table per sequence;
allocate on demand, release on EOS
fragmentation < 4%
vs 60-80% contiguous
pay only for tokens used
--gpu-memory-utilization = 0.9
the knob (not a flag)
PagedAttention is the only
allocator vLLM ships
logical to physical map
seq_A: [tok0..15] to block 37
seq_A: [tok16..31] to block 12
seq_B: [tok0..15] to block 42
... block table grows with seq
KV is virtual memory for sequences
continuous batching
iteration-level scheduling
admit / release every decode step
the loop
1. drop finished seqs from RUNNING
2. admit WAITING if KV blocks free
3. one forward across the batch
batch is always full of real work
no padding to longest seq
fast seqs leave, new seqs join
V1 scheduler (2026)
scheduler runs per iteration
not per request
check feature pairs against
the version's compat matrix
3-4x over naive at 128 concurrent
chunked prefill
slice prefill into 512-tok chunks
interleave with decode
protect TTFT tail, not mean
P99 ITL under mixed load
without chunking: ~50 ms
with chunking: ~15 ms
vLLM v0.18.0 feature matrix
spec decode + chunked prefill
marked compatible
known spec-decode issues
pipeline parallelism:
vllm <= 0.15.0
draft models: vllm <= 0.10.0
read the release notes before flipping flags
all three on together — Llama 3.3 70B FP8, H100 SXM5, 128 concurrent
throughput 2,200-2,400 tok/s · 25% over default vLLM · 3-4x naive PyTorch · < 4% fragmentation
each default assumes the other two