Quantization formats — pick by hardware, engine, workload
the picking tree
CPU / edge ? → GGUF Q4_K_M (llama.cpp)
Reasoning workload ? → FP8 (vLLM or TRT-LLM)
Multi-LoRA on vLLM ? → GPTQ-Int4 + Marlin
Routine chat, GPU ? → AWQ-Int4 + Marlin
Blackwell, quality validated ? → NVFP4 + FP8 KV
trap #1 — the KV cache
AWQ shrinks weights, not KV
70B AWQ weights : ~35 GB
KV at 128 conc / 2k ctx : ~20-30 GB
budget HBM holistically
weights + KV + activations
FP8 KV buys equivalent savings to AWQ weights
KV quantization is a separate choice
affects attention accuracy directly
trap #2 — calibration dataset
AWQ and GPTQ calibrate on C4/Wiki by default
for domain models : wrong weights get protected
Pass@1 on HumanEval can drop 3-5 points
fix : calibrate on in-domain data
few hundred samples usually enough
always test on eval set before shipping
AWQ INT4 is hazardous for reasoning : use FP8
numbers (2026)
GGUF Q4_K_M on vLLM : ~93 tok/s (wrong engine)
GPTQ-Int4 + Marlin on vLLM : ~712 tok/s
AWQ-Int4 + Marlin : ~741 tok/s ; best Pass@1 INT4
FP8 on H100 : ~2x BF16 throughput
NVFP4 on Blackwell : ~4x BF16 ; reasoning risk