An FP8 KV cache holds twice the tokens in each GiB
Computed KV cache tokens that fit in one GiB for Qwen3-8B, which has 36 layers, 8 KV heads and a head size of 128. BF16 uses 2 bytes per value, 144 KiB per token, and holds 7,281 tokens per GiB. FP8 uses 1 byte per value, 72 KiB per token, and holds 14,563 tokens per GiB. These numbers come from the formula, not from a run.
0
6,000
12,000
18,000
KV tokens per GiB
7,281
BF16, 144 KiB per token
14,563
FP8, 72 KiB per token
Computed from the bytes per token. Our runs held the block count fixed.