Inference metrics — throughput lies, goodput tells the truth
the four latencies
TTFT = queue + network + prefill
TPOT / ITL = decode per token after first
E2E = TTFT + TPOT * output_len
throughput = total_tokens / elapsed
throughput is a fleet metric, not a user metric
goodput — the SLO gate
goodput = P[ all SLO constraints held ]
TTFT P99 <= a AND
TPOT P99 <= b AND
E2E P99 <= c
a request is "good" only if every bound held
target : goodput >= 99%
used in MLPerf Inference v6.0 submissions
percentiles — never mean alone
always P50, P90, P99
LLM distributions are right-skewed
mean TPOT 7 ms can hide P99 of 40 ms
reference (Llama-3.1-8B on TRT-LLM, 2026)
TTFT 162 ms, TPOT 7.33 ms, E2E 1093 ms (mean)
measurement trap
GenAI-Perf vs LLMPerf
same run, different TPOT
GenAI-Perf (NVIDIA)
ITL excludes TTFT ; starts from token 2
LLMPerf (Ray / Anyscale)
ITL includes TTFT ; starts from token 1
state tool + version in every report