vLLM production-stack + LMCache — cross-engine KV reuse
router (cache-aware)
consumes KV events, hashes prefixes, routes to engine with local OR shared match
vLLM engine 1
H100 80GB
Connector API v0.9+
0.11.0 async offload
vLLM engine 2
H100 80GB
vLLM engine 3
H100 80GB
vLLM engine 4
H100 80GB
LMCache — cluster-shared KV cache
CPU DRAM tier (512 GB+ per socket)
asynchronous via Connector API; offload is not user-facing
decisive for: multi-tenant shared prompts, RAG chunk reuse, LoRA on shared base, preemption restore
durable tier — Ceph / S3 / local NVMe
for blocks evicted from DRAM; slower but infinite
16x H100 benchmark: LMCache helps when KV footprint exceeds HBM; matches baseline when it does not