cache-aware multi-region routing — TTFT vs network RTT
incoming request
origin: us-east-1
prefix_hash = 0xabc
cache-aware router (vLLM Router / llm-d)
1. hash prefix → 0xabc
2. lookup replicas holding 0xabc
3. minimize prefill + network RTT
KV events: cache.block_added / cache.block_evicted
choice
replica R3 (us-east)
hit=true, local
TTFT ≈ 80 ms
vs ~800 ms cold
us-east-1 (origin)
R1 R2 R3* R4
R3 holds prefix 0xabc
RTT 0 ms
local cache hit: 80 ms
us-west-2
R5 R6 R7 R8
no match on 0xabc
RTT 65 ms
cold prefill: 800 + 65 ms
eu-west-1
R9 R10 R11 R12
R11 holds prefix 0xabc
RTT 75 ms
hit + cross: 80 + 75 ms
routing objective — GORGO (2026)
minimize: prefill_time + network_RTT
· cache hit saves ~720 ms
· cross-region costs ~65-220 ms RTT
· cross-region wins when prefill dominates
· residency boundary beats both — partition first
DR manifest — the 32% failure
weights backed up, tokenizer was not
required files:
· tokenizer.json / tokenizer.model
· quantization_config.json, AWQ scales
· RoPE scaling, chat template, LoRA adapters