The pool in tokens decides how many users fit
On the L4 the pool held 24,096 tokens. The startup line reports 2.94 users because it assumes every user fills the full 8,192-token context. With 512 prompt tokens and 128 output tokens, each user holds 640 tokens, so 37 users fit.
KV POOL ON THE L4, QWEN3-8B
24,096 tokens, 3.31 GiB at 144 KiB each
IF EVERY USER FILLS 8,192 TOKENS
Maximum concurrency 2.94x, the number the startup line prints
IF EACH USER HOLDS 640 TOKENS
37 users fit, each with 512 prompt and 128 output tokens