two-layer caching — L1 skips LLM, L2 reuses KV
L1 — semantic (app-level)
embed prompt → find similar cached
tools: GPTCache, Redis Vec, Portkey, Helicone
threshold: cosine ≥ 0.95 (start), tune
hit rates in production:
open-ended chat: 10-15%
structured FAQ: 40-70%
code questions: 20-30%
voice agent repeat: 50-80%
vendor "95% accuracy" = match correctness, not hit rate
L2 — provider prefix cache
reuse attention KV for repeated prefix
Anthropic: explicit cache_control marker
cache read $0.30/M vs $3.00/M (10x)
write premium: 1.25x (5-min TTL) or 2x (1-hr TTL)
OpenAI: automatic, prompts > 1024 tokens
cached input ~10x cheaper; ~50% typical hit
Gemini: explicit context caching API
Self-hosted: Phase 17 · 06 RadixAttention
ProjectDiscovery: 7% → 74% hit rate with prompt restructure
anti-patterns that collapse hit rate
· parallelization penalty
10 parallel calls before first cache write = 10 writes, 0 reads → 5-10x bill
· dynamic content in prefix
current time, request ID, randomized examples: every request is unique → 0 hits
stack for overnight
Batch API (50% off) + cached input (10x on input) → ~10% of sync-uncached cost
anything tolerant of 24-hour latency should stack both