Open-weight VLM design space — five axes and their benchmark pull
the five axes of every open VLM recipe
1. image encoder
CLIP / SigLIP 2
DINOv2 / InternViT
concat for dense
MM1: +3 MMMU
SigLIP 2 wins 2026
2. connector
MLP / Q-Former
Perceiver / C-Abstr
SVA (Cambrian)
~1 pt between them
picks token count
3. LLM
Llama-3 / Mistral
Qwen2.5 / Phi
7B -> 70B plateau
+2-4 / doubling
sets reasoning ceiling
4. data
caption pairs
interleaved OBELICS
instr: LLaVA/ShareGPT4V
PixMo human caps
detail > distill
5. resolution
fixed 336/448
AnyRes tiling
native dynamic
ramped schedule
OCR multiplier
Prismatic VLMs variance decomposition (controlled 7B comparison)
visual-token count
~60%
image encoder
~20%
LLM size
~15%
data mix
~10%
connector arch
~5%
resolution sched
~5%
2026 default recipe
encoder: SigLIP 2 SO400m/14
connector: 2-layer MLP
LLM: Qwen2.5-7B (or 70B)
data: PixMo + Cauldron
res: dynamic 256-1280
schedule: 3 stages
ablate first:
1. token count
2. encoder
3. data mix