KV cache grows. Flash Attention keeps the hot path in SRAM.
KV cache — decoding step by step
t=1
k1
v1
t=3
k1
k2
k3
v1
v2
v3
t=N
each new query attends to N cached k/v — O(N) per step, not O(N²)
Flash Attention — tiling into SRAM
HBM
3 TB/s, 80 GB
Q
K
V
O
no N×N ever materialized here
SRAM (per SM)
30 TB/s, 256 KB
Qt
Kt
Vt
Ot
running (max, sum)
softmax exact, not approximated
Flash Attention version timeline
FA1 (2022)
SRAM tiling
2× on A100
FA2 (2023)
better parallelism
3× on A100
FA3 (2024)
Hopper asynchrony, FP8
~740 TFLOPs H100
FA4 (2026)
Blackwell 5-stage pipe
inference-first
next
backward + GQA
varlen for training
KV cache and Flash Attention are universal in 2026 — vLLM, SGLang, TensorRT-LLM, llama.cpp all assume both.
Next frontier: PagedAttention (KV as virtual memory), prefix caching, speculative decoding.