The bytes one token holds in the KV cache
Bytes per token are layers times KV heads times head dimension times two, for keys and values, times two bytes for BF16. Qwen3-8B has 36 layers, 8 KV heads and dimension 128, so 144 KiB. Qwen3-30B-A3B has 48 layers and 4 KV heads, so 96 KiB.
LAYERS
KV HEADS
HEAD DIM
K AND V
BF16 BYTES
MODEL
Qwen3-8B
36
x
8
x
128
x
2
x
2
144 KiB for every token of every user
Qwen3-30B-A3B
48
x
4
x
128
x
2
x
2
96 KiB for every token of every user