The same feature at two batch sizes
Two panels of output tokens per second for Llama-3.1-8B on one A6000 from issue 5274. At batch 1, without speculation 42.0 and with EAGLE-3 92.5. At concurrency 20, without speculation 217.8 and with EAGLE-3 164.4. Both panels use one vertical scale.
A · BATCH 1
B · CONCURRENCY 20
42.0
no speculation
92.5
EAGLE-3
217.8
no speculation
164.4
EAGLE-3
output tok/s, issue 5274, 2025-04-11