Fixed lengths trade a slow first token for smooth decoding
Median time to first token and P99 inter-token latency against --random-range-ratio on one NVIDIA L4 with Qwen3-8B, a cap of 16 and 64 prompts, three seeds per point, on a log scale. At a ratio of 1.0 every request has 512 input and 128 output tokens, and the median TTFT is 1,996 ms for BF16 and 1,526 ms for FP8, while P99 ITL is 82 and 55 ms. At a ratio of 0.0 the lengths vary, the median TTFT is 291 and 129 ms, and P99 ITL is 300 and 95 ms.
30
100
300
1k
3k
0
0.5
1
--random-range-ratio (1.0 means every request has the same length)
ms (log scale)
BF16 median TTFT
FP8 median TTFT
BF16 P99 ITL
FP8 P99 ITL