What each line of the benchmark report is computed from
Five lines of the benchmark report, each with the formula calculate_metrics uses and the value from the run at a cap of 8: Concurrency 7.96, Peak concurrent requests 16, median TTFT 785.79 ms, median TPOT 22.52 ms, median ITL 20.35 ms.
REPORT LINE
COMPUTED AS
AT CAP 8
Concurrency
sum of E2E latencies / duration
7.96
Peak concurrent requests
most requests active in one second
16
TTFT
send to first streamed chunk
785.79 ms
TPOT
(latency - TTFT) / (tokens - 1)
22.52 ms
ITL
every gap between two chunks, pooled
20.35 ms
Median values for TTFT, TPOT, and ITL.