Thomas Piacentini
← Workshop

Cheat sheet

Inference Metrics and Load Testing

Original text · English

Find why a load test lies. TTFT, TPOT, goodput, P99, realistic prompts, retry storms and hedged reads.

Symptoms, causes, fixes

SymptomCauseFix
Test held at 500 users, prod fell at 200Identical prompts hit the prefix cache--mean-input-tokens plus --stddev-input-tokens
Locust ITL high, server is fineGIL trap: client tokenization backlogLLM-Locust or a compiled harness
GenAI-Perf TPOT 6 ms, LLMPerf 11 msGenAI-Perf excludes TTFT from ITLState the tool and the definition
High throughput, users leaveLow goodput, tail blows the SLOGate deploys on goodput
Mean TPOT 9 ms, users complainRight-skewed tail, P99 TPOT 65 msReport P50, P90, P99
TPOT spikes to 50 msLong prefill on a neighbor sequenceChunked prefill
All clients retry at the same timeSynchronized exponential waits--jitter=true and a shared --budget
Hedge looks freeCancel does not stop server workCount fixture work next to latency
Sources

Metric formulas

TTFT = queue_time + network_request + prefill_timebefore anything streams
TPOT = (decode_forward_time + scheduler_overhead) / tokens_producedsame as ITL
E2E = TTFT + TPOT * output_tokens + network_responseTPOT-dominated above 500 tokens
throughput = total_output_tokens / elapsed_timefleet efficiency only
goodput = fraction of requests meeting (TTFT <= a) AND (TPOT <= b) AND (E2E <= c)
Sources

SLO profiles in main.py

Factorloosetargettight
TTFT ms800500300
TPOT ms251510
E2E ms300020001500
VerdictSHIPPABLE at 99%DEGRADED at 95%FAILING below
Sources

Four load patterns

FactorSteady-stateRampSpikeSoak
ShapeConstant RPS0 to target RPSSudden 3-10x RPS, then backSteady-state
Duration30-60 min15 min2 min4-8 hours
CatchesBaseline regressionsCapacity breakpoint, warm-up anomaliesAutoscaling latency, queue saturation, cold startMemory leaks, connection-pool drift
Sources

Use each tool when

LLMPerf
a benchmark run. Rust-backed tokenization, mean/stddev prompts.
GenAI-Perf
you need the NVIDIA reference. ITL excludes TTFT.
LLM-Locust
your team uses Locust. It fixes the GIL trap.
guidellm
large-scale synthetic benchmarking.
k6 v2026.1.0
CI/CD gates and SLA tests. Streaming-aware, no GIL.
k6 Operator
distributed tests through TestRun / PrivateLoadZone CRDs.
Vegeta
constant-rate saturation of a gateway or rate limit.
Locust 2.43.3 stock
never alone for LLMs. It has the GIL trap.
Sources

Run the labs

python3 scripts/project_test.py retry-storm-lab --init /tmp/retry-storm-lab-work
python3 scripts/project_test.py retry-storm-lab --all --solution --strict
cd projects/retry-storm-lab/solution
go run . --clients 20 --budget 60 --jitter=false --out /tmp/synchronized.jsonsynchronized waits
go run . --clients 20 --budget 60 --jitter=true --seed 7 --out /tmp/jittered.jsonseeded spread
go run . --local-http --clients 20 --budget 12 --out /tmp/retry-wire.jsonreal loopback HTTP
node --experimental-strip-types cli.ts /tmp/retry-wire.jsonverify counters
cd projects/hedged-request-lab/solution
go run . --samples 3 --delay-ms 10 --out /tmp/hedge-report.json80ms primary, 5ms secondary
Sources

Lab flag defaults

SettingDefaultNote
--clients20Concurrent clients
--budget12Shared retry attempts
--base-ms10Initial backoff
--max-ms80Maximum retry wait
--attempts4Maximum attempts per client
--capacity5Simulated capacity per exact millisecond
--delay-ms10Hedge delay
--samples3Samples per policy
Sources

Lab error strings

CommandWhat it does
invalid bounded policyBase below 1, cap below base, or attempts over 10
non-retryable HTTP statusStatus is not 429, 502, 503 or 504
server delay exceeds policy capRetry-After is above the policy cap
retry attempts or shared budget exhaustedNo attempts or budget left
winner requires HTTP 200A fast error cannot win the hedge
body exceeds 64 KiBResponse body is over the read limit
no valid responseBoth hedge attempts failed
samples must be 1..100Sample count is out of range
Sources

One hedged read

  1. Client starts the primary GET and a delay timer.
  2. Client a valid primary before the timer prevents the hedge.
  3. Client else launches at most one secondary. Delay is 0 to 1 second.
  4. Validator only HTTP 200 with a nonempty JSON value can win.
  5. Client cancels the loser and observes it for at most 100ms.
  6. Receipt records late and unobserved attempts and winner latency.
  7. Fixture counts server work that continues after cancel.
Sources

Worked numbers

TTFT 500 ms, 100 tokens in 700 ms decode?
GenAI-Perf ITL = 700/99 = 7.07 ms. LLMPerf ITL = 1200/100 = 12.00 ms.
Twenty clients, four attempts, budget 12?
At most 32 total calls.
Base 10, cap 25, no jitter?
Waits of 10,20,25,25.
Three baseline and three hedged samples?
Three calls versus six. The fixture records nine.
Samples [2,4,10,20,80]?
P50=10, P95=80, maximum=80.
Primary HTTP 500 at 1ms, secondary valid at 15ms?
The secondary wins. A fast error cannot win.
Sources

Reference numbers and gates

  • Llama-3.1-8B-Instruct on TRT-LLM: mean TTFT 162 ms, mean TPOT 7.33 ms.
  • Consumer 70B SLO: TTFT P99 <= 800 ms, TPOT P99 <= 25 ms, goodput >= 99%.
  • Run at 2x peak concurrency. Take 30-50 iterations.
  • Use --mean-input-tokens 800 --stddev-input-tokens 300 --mean-output-tokens 150.
  • CI gate: P50/P95 TTFT, 5xx < 5%, TPOT under threshold. Break the build on breach.
  • Publish tool name, tool version, model, hardware, concurrency and prompt distribution.
Sources
Original source
---
id: inference-load-testing
title: Inference Metrics and Load Testing
code: IM
subtitle: Find why a load test lies. TTFT, TPOT, goodput, P99, realistic prompts, retry storms and hedged reads.
hue: 352
sources: phases/17-infrastructure-and-production/08-inference-metrics-goodput, phases/17-infrastructure-and-production/22-load-testing-llm-apis, projects/retry-storm-lab, projects/hedged-request-lab
source_hash: 26e858df4a14ef8f
---

## Symptoms, causes, fixes
type: troubleshooting
from: phases/17-infrastructure-and-production/08-inference-metrics-goodput, phases/17-infrastructure-and-production/22-load-testing-llm-apis, projects/retry-storm-lab, projects/hedged-request-lab

| Symptom | Cause | Fix |
|---|---|---|
| Test held at 500 users, prod fell at 200 | Identical prompts hit the prefix cache | `--mean-input-tokens` plus `--stddev-input-tokens` |
| Locust ITL high, server is fine | GIL trap: client tokenization backlog | LLM-Locust or a compiled harness |
| GenAI-Perf TPOT 6 ms, LLMPerf 11 ms | GenAI-Perf excludes TTFT from ITL | State the tool and the definition |
| High throughput, users leave | Low goodput, tail blows the SLO | Gate deploys on goodput |
| Mean TPOT 9 ms, users complain | Right-skewed tail, P99 TPOT 65 ms | Report P50, P90, P99 |
| TPOT spikes to 50 ms | Long prefill on a neighbor sequence | Chunked prefill |
| All clients retry at the same time | Synchronized exponential waits | `--jitter=true` and a shared `--budget` |
| Hedge looks free | Cancel does not stop server work | Count fixture work next to latency |

## Metric formulas
type: annotated-code
lang: text
span: 2
from: phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md

```text
TTFT = queue_time + network_request + prefill_time  # before anything streams
TPOT = (decode_forward_time + scheduler_overhead) / tokens_produced  # same as ITL
E2E = TTFT + TPOT * output_tokens + network_response  # TPOT-dominated above 500 tokens
throughput = total_output_tokens / elapsed_time  # fleet efficiency only
goodput = fraction of requests meeting (TTFT <= a) AND (TPOT <= b) AND (E2E <= c)
```

## SLO profiles in main.py
type: comparison
from: phases/17-infrastructure-and-production/08-inference-metrics-goodput

| Factor | loose | target | tight |
|---|---|---|---|
| TTFT ms | 800 | 500 | 300 |
| TPOT ms | 25 | 15 | 10 |
| E2E ms | 3000 | 2000 | 1500 |
| Verdict | SHIPPABLE at 99% | DEGRADED at 95% | FAILING below |

## Four load patterns
type: comparison
span: 2
from: phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md

| Factor | Steady-state | Ramp | Spike | Soak |
|---|---|---|---|---|
| Shape | Constant RPS | 0 to target RPS | Sudden 3-10x RPS, then back | Steady-state |
| Duration | 30-60 min | 15 min | 2 min | 4-8 hours |
| Catches | Baseline regressions | Capacity breakpoint, warm-up anomalies | Autoscaling latency, queue saturation, cold start | Memory leaks, connection-pool drift |

## Use each tool when
type: concepts
from: phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md

- **LLMPerf**: a benchmark run. Rust-backed tokenization, mean/stddev prompts.
- **GenAI-Perf**: you need the NVIDIA reference. ITL excludes TTFT.
- **LLM-Locust**: your team uses Locust. It fixes the GIL trap.
- **guidellm**: large-scale synthetic benchmarking.
- **k6 v2026.1.0**: CI/CD gates and SLA tests. Streaming-aware, no GIL.
- **k6 Operator**: distributed tests through TestRun / PrivateLoadZone CRDs.
- **Vegeta**: constant-rate saturation of a gateway or rate limit.
- **Locust 2.43.3 stock**: never alone for LLMs. It has the GIL trap.

## Run the labs
type: annotated-code
lang: bash
span: 2
from: projects/retry-storm-lab/README.md, projects/hedged-request-lab/README.md

```bash
python3 scripts/project_test.py retry-storm-lab --init /tmp/retry-storm-lab-work
python3 scripts/project_test.py retry-storm-lab --all --solution --strict
cd projects/retry-storm-lab/solution
go run . --clients 20 --budget 60 --jitter=false --out /tmp/synchronized.json  # synchronized waits
go run . --clients 20 --budget 60 --jitter=true --seed 7 --out /tmp/jittered.json  # seeded spread
go run . --local-http --clients 20 --budget 12 --out /tmp/retry-wire.json  # real loopback HTTP
node --experimental-strip-types cli.ts /tmp/retry-wire.json  # verify counters
cd projects/hedged-request-lab/solution
go run . --samples 3 --delay-ms 10 --out /tmp/hedge-report.json  # 80ms primary, 5ms secondary
```

## Lab flag defaults
type: defaults
from: projects/retry-storm-lab/solution/main.go, projects/retry-storm-lab/README.md, projects/hedged-request-lab/solution/main.go, projects/hedged-request-lab/README.md

| Setting | Default | Note |
|---|---|---|
| `--clients` | 20 | Concurrent clients |
| `--budget` | 12 | Shared retry attempts |
| `--base-ms` | 10 | Initial backoff |
| `--max-ms` | 80 | Maximum retry wait |
| `--attempts` | 4 | Maximum attempts per client |
| `--capacity` | 5 | Simulated capacity per exact millisecond |
| `--delay-ms` | 10 | Hedge delay |
| `--samples` | 3 | Samples per policy |

## Lab error strings
type: command-table
from: projects/retry-storm-lab/solution/main.go, projects/hedged-request-lab/solution/main.go

| Command | What it does |
|---|---|
| `invalid bounded policy` | Base below 1, cap below base, or attempts over 10 |
| `non-retryable HTTP status` | Status is not 429, 502, 503 or 504 |
| `server delay exceeds policy cap` | Retry-After is above the policy cap |
| `retry attempts or shared budget exhausted` | No attempts or budget left |
| `winner requires HTTP 200` | A fast error cannot win the hedge |
| `body exceeds 64 KiB` | Response body is over the read limit |
| `no valid response` | Both hedge attempts failed |
| `samples must be 1..100` | Sample count is out of range |

## One hedged read
type: flow
from: projects/hedged-request-lab/stages, projects/hedged-request-lab/README.md

1. **Client**: starts the primary GET and a delay timer.
2. **Client**: a valid primary before the timer prevents the hedge.
3. **Client**: else launches at most one secondary. Delay is 0 to 1 second.
4. **Validator**: only HTTP 200 with a nonempty JSON value can win.
5. **Client**: cancels the loser and observes it for at most 100ms.
6. **Receipt**: records late and unobserved attempts and winner latency.
7. **Fixture**: counts server work that continues after cancel.

## Worked numbers
type: qa
from: phases/17-infrastructure-and-production/08-inference-metrics-goodput, projects/retry-storm-lab/stages, projects/hedged-request-lab/stages

- **TTFT 500 ms, 100 tokens in 700 ms decode?** GenAI-Perf `ITL = 700/99 = 7.07 ms`. LLMPerf `ITL = 1200/100 = 12.00 ms`.
- **Twenty clients, four attempts, budget 12?** At most 32 total calls.
- **Base 10, cap 25, no jitter?** Waits of 10,20,25,25.
- **Three baseline and three hedged samples?** Three calls versus six. The fixture records nine.
- **Samples [2,4,10,20,80]?** P50=10, P95=80, maximum=80.
- **Primary HTTP 500 at 1ms, secondary valid at 15ms?** The secondary wins. A fast error cannot win.

## Reference numbers and gates
type: tips
from: phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md, phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md

- Llama-3.1-8B-Instruct on TRT-LLM: mean TTFT 162 ms, mean TPOT 7.33 ms.
- Consumer 70B SLO: TTFT P99 <= 800 ms, TPOT P99 <= 25 ms, goodput >= 99%.
- Run at 2x peak concurrency. Take 30-50 iterations.
- Use `--mean-input-tokens 800 --stddev-input-tokens 300 --mean-output-tokens 150`.
- CI gate: P50/P95 TTFT, 5xx < 5%, TPOT under threshold. Break the build on breach.
- Publish tool name, tool version, model, hardware, concurrency and prompt distribution.
Code and artifacts 1
What you take away

Saved in your notebook on this device.

Sources and limits

Reading checkpoints are self-reported. Test results are recorded separately and apply to the executed cases.

AI Engineering from Scratch · 3f9715c · MIT

Download this file ↓