workload triage — three lanes, one rule
interactive
user waits for response
· TTFT matters
· sync call with prompt cache
· cannot batch
· examples: chat, voice agent, search
100% of baseline cost
(minus caching savings)
semi-interactive
user checks back in minutes
· async queue + fallback to sync
· tolerate P99 ~5-10 min
· examples: moderate RAG index,
"refresh" report, semi-live agent
~50% of baseline cost
(with caching + queue amortization)
batch
result "by morning"
· always batch, always stack cache
· 24h SLA, typical 2-6h P50
· examples: content pipelines,
classification, labeling, reports
~10% of baseline cost
(batch 50% off + cached input 10x)
the rule
if 24h latency is acceptable to the user, the workload belongs on batch