Throughput stops at a cap of 8, latency does not
Two bar charts from the benchmark sweep. Left, output token throughput at caps 1, 8 and 32 is 175.15, 255.63 and 252.74 tokens per second. Right, median time to first token at the same caps is 162.98, 785.79 and 3917.54 milliseconds.
OUTPUT TOK/S
MEDIAN TTFT, MS
cap 1
175.15
cap 1
163
cap 8
255.63
cap 8
786
cap 32
252.74
cap 32
3918
+46% from 1 to 8, -1% from 8 to 32
x4.8 from 1 to 8, x5.0 from 8 to 32