Prefill against decode on the capture Mac
Two columns from one benchmark run with one request at a time. On the left, prefill runs one forward pass over 512 prompt tokens, writes 512 KV entries, and returns the first token after a median of 162.98 ms, about 0.32 ms per prompt token. On the right, decode runs one forward pass for each of the next 127 tokens, reads every weight and every cached KV entry each time, and takes a mean of 4.38 ms per token.
PREFILL · THE WHOLE PROMPT AT ONCE
DECODE · ONE TOKEN PER PASS
512 prompt token ids
random dataset, seed 7
one forward pass
512 tokens in one batch of matmuls
512 KV entries written
kept for every later step
median TTFT 162.98 ms
about 0.32 ms for each prompt token
1 new token per request
the token decode made last
one forward pass
reads every weight for one token
reads 512 + n KV entries
and writes one more
mean TPOT 4.38 ms
repeated for each of the next 127 tokens
Qwen3-0.6B-4bit · SGLang 0.5.22 on MLX