Why five requests fit in 4,096 tokens and six do not
Two bars show a KV pool of 4,096 tokens. In the first, five requests each reserve 256 prompt tokens and 512 output tokens, 3,840 tokens in all, with 256 tokens free. In the second, a sixth request would need 768 more tokens and runs 512 tokens past the end of the pool, so the scheduler keeps it waiting.
pool end
ADMITTED: FIVE REQUESTS
each request reserves 256 prompt + 512 output = 768 tokens
A SIXTH REQUEST WOULD NEED 4,608
0
3,840
4,096
prompt KV
reserved output
free
does not fit