BLIP-2 Q-Former bridge — 32 queries compress the ViT for a frozen LLM
frozen ViT
ViT-g/14, 1.1B
224x224 input
16x16 patch grid
-> 256 patch tokens
dim 1408
not trained
outputs frozen features
256 x 1408 per image
K, V
Q-Former (trained)
12 layers, BERT-base init
32 learnable queries
parameters of the bridge
same 32 vectors for every image
self-attention (queries only)
queries interact with each other
and with text in stage 1
cross-attention
Q from queries, K/V from patches
32 x 256 attention map
FFN + LN (shared with text path)
32 tokens
linear projection
768 -> 4096 (LLM dim)
trained in stage 2
~3M params
32 x LLM_dim output
frozen LLM
OPT-6.7B or
Flan-T5-XXL
32 visual tokens
+ text prompt
-> caption / VQA
not trained
LM generation
caption or answer
two-stage training
Stage 1: representation (no LLM)
train Q-Former alone with
ITC (image-text contrastive)
+ ITM (image-text matching)
+ ITG (image-grounded text gen)
queries learn to encode
both semantic and text-decodable info
Stage 2: generative (with frozen LLM)
project 32 queries -> LLM dim
prepend to text
train LM loss end-to-end
bridge + projector learn
to feed the LLM cleanly
188M params total