FinOps for LLMs — three dimensions, four layers, three-tier enforcement
three attribution dimensions
per-user (user_id)
per-task (task_id + route)
per-tenant (tenant_id)
instrument all three on day 1
four token layers
· prompt (40-60%)
· tool (20-40%, agent-heavy)
· memory (10-30%, RAG)
· response (10-30%)
bucketing all four hides the optimization lever
three-tier enforcement
1. rate limit per tenant
2-3x peak, 429 + Retry-After
2. daily spend cap per tenant
1.5-3x contract; tighten rate + alert CS
3. kill switch on z-score > 4
auto-pause + page on-call
unit metric — cost per product outcome, not $/M tokens
cost per resolved support ticket
cost per generated article
cost per successful agent task
cost per user-session-minute
cost per index refresh
tie cost to a product outcome, else optimization is unanchored
stacked optimization ceiling — ~5-10% of naive baseline
cache (14) + batch (15) + route (16) + gateway (19)
each halves or 10xs; compound multiplies
most teams run 2-3 levers; few stack all four
leaked spend = gap between best-case stack and current