Durable execution for multi-agent systems
super-step + checkpoint pattern (LangGraph, Temporal, Restate)
super-step 1
plan
write ckpt
super-step 2
research
write ckpt
super-step 3
CRASH
resume
from ckpt 2
another worker
super-step 3'
research (re-done)
idempotent tool calls
super-step 4
synthesize + reply
tier the choice to the load
Tier 1 (Bedi's rule)
FastAPI + Postgres + outbox
< 100 concurrent runs, sub-hour tasks
Tier 2
LangGraph runtime or Temporal
hour-long runs, interrupt/resume, retries
Tier 3
custom outbox + event sourcing, specialized
async vs threads for I/O-bound LLM calls
asyncio / fibers
cooperative yield on I/O
10k concurrent: fine
thread-per-call
~1MB stack each
10k concurrent: ~10GB
LLM calls idle 99% of wall-clock on I/O; threads pay memory for no gain
CPU-bound post-processing stays on threads / processes
rainbow deployments keep long-running agents alive across code pushes