Flamingo — gated cross-attention inside a frozen 70B Chinchilla LLM
image / video pipeline
frozen ViT
196-900 patches / image
Perceiver resampler
K=64 learnable latents
cross-attn over patches
6 blocks, 64M params
fixed 64 tokens out
64 visual tokens
per image (or per video)
variable inputs
image: 196 patches -> 64
high-res: 900 patches -> 64
video: T * 64
output shape: always K=64
frozen LLM with inserted gated cross-attn
LLM self-attn + FFN (block 1-4)
frozen Chinchilla 70B layers
GATED CROSS-ATTENTION (trained)
y = tanh(alpha) * cross_attn(text, visual_tokens) + x
alpha init 0 -> tanh(0) = 0 -> output == input
-> LLM text capability preserved at step 0
LLM self-attn + FFN (block 5-8)
text tokens continue causal self-attention
GATED CROSS-ATTENTION (trained)
as alpha increases, visual contribution mixes in
ends training with alpha O(1-2), tanh ~0.8
inserted every M=4 LLM layers
LLM self-attn + FFN (block 9-12, ..., N)
total cross-attn layers: N/4
for 70B LLM with 80 layers -> 20 gated bridges
trained params: ~10B / 80B total