chaos engineering for LLMs — four planes + feedback loop
control plane
experiment scheduler
· LitmusChaos workflow
· Chaos Mesh CRDs
· Harness UI
· AWS FIS / Azure Chaos Studio
schedules, parameters, cadence
target plane
services + infra + data
· pods, nodes, LB
· data stores, vector DB
· LLM gateways
· upstream providers
safety plane
guards + abort
· burn-rate gate (> 2x budget)
· blast-radius cap
· suppression windows
· trace-ID tagging
· kill switch
observability
traces + metrics
· distinguish
chaos-induced from
natural failures
· trace-ID correlation
five LLM-specific experiments
1. memory overload (KV preemption storm)
2. network failure (cut gateway ↔ provider)
3. provider outage (100% 429)
4. malformed prompt (tokenizer bomb)
5. KV eviction storm (force cascading re-prefill)
cadence
weekly canary + SLO review · monthly game day + postmortem · quarterly audit + dep map
without SLI/SLO + observability + rollback + runbooks + on-call, don't run in prod