LLaVA recipe — simpler projector, cheaper training, bigger ecosystem
forward pass: image -> projector -> LLM
CLIP ViT-L/14
@ 336x336
frozen stage 1
303M params
576 patch tokens
dim 1024 each
24x24 grid
2-layer MLP projector
1024 -> 4096 -> 4096
GELU activation
~22M params
shared across all patches
trained in stage 1
fine-tuned in stage 2
the whole bridge
576 visual tokens
each dim 4096
replaces <image>
in the text prompt
token budget:
576 / 2048 = 28%
576 / 32k = 1.8%
AnyRes: 2880 tokens
Vicuna 13B
or Llama-3.1
frozen stage 1
trained stage 2
autoregressive
next-token
generation
same as text-only
two-stage training
Stage 1: projector alignment
freeze ViT, freeze LLM
train ONLY the 2-layer MLP
data: 558k caption pairs (LAION-CC-SBU)
loss: LM on caption, conditioned on visual tokens
cost: ~4 hours on 8xA100
goal: projector learns ViT-space -> LLM-space
no task-specific objectives
Stage 2: visual instruction tuning
unfreeze LLM (full or LoRA)
keep projector trainable
data: 158k GPT-4-generated turns
(conversation / description / reasoning)
cost: ~20 hours on 8xA100
goal: dialog, VQA, instruction following
LLaVA-NeXT adds ShareGPT4V, academic mix