load testing LLM APIs — two traps, four patterns, five tools
GIL trap (Locust stock)
· client tokenizes under Python GIL
· competes with request generation
· tokenization backlog inflates reported ITL
your client is the bottleneck, not the server
prompt-uniformity trap
· loop with one prompt = 100% prefix cache
· request coalescing serves "N concurrent" as 1
· throughput looks great, production falls over
fix: LLMPerf --mean + --stddev input tokens
2026 tools
LLMPerf — Anyscale, Rust tokenizers, streaming
NVIDIA GenAI-Perf — Triton-backed reference
LLM-Locust — Locust + GIL fix
guidellm — large-scale synthetic
k6 v2026.1.0 + Operator 1.0 GA
streaming-aware, CRD-native, best CI gate
four load patterns
steady — 30-60 min constant RPS
catches baseline regressions
ramp — 0 to target over 15 min
catches capacity breakpoint + warm-up
spike — 3-10x sudden burst
catches autoscaling + cold-start impact
soak — 4-8h steady
CI gate recipe
k6 on PR with 30-50 iterations at baseline RPS
gate: P50 / P95 TTFT, 5xx < 5%, TPOT threshold
break the build on breach — treat performance as a compile error
GenAI-Perf ITL excludes TTFT · LLMPerf includes it — same server, different TPOT