InternVL3 — native multimodal pretraining vs post-hoc adaptation
post-hoc (LLaVA, Qwen-VL v1, Idefics)
pretrained LLM (frozen)
vision encoder
projector
1. train projector on caption pairs (LLM frozen)
2. unfreeze LLM, tune on LLaVA-Instruct
3. optional task fine-tune
cost: ~30k GPU-hours, reuses LLM
alignment debt: -2 to -8 MMLU, answer drift
native (InternVL3, Chameleon, GPT-4o)
single transformer, text + vision native from step 1
one pretraining run, one loss
40% text + 35% interleaved + 20% cap + 5% video
instruction tune after, not multi-stage alignment
cost: ~300k GPU-hours, no LLM reuse
no alignment debt, matches Gemini 2.5 Pro at 78B
deployment optimizations: ViR + DvD
Visual Resolution Router (ViR)
small classifier picks min resolution per query
low / medium / high tiers
50% of real queries are low-res candidates
avg tokens drop from 2048 to ~590 per query
net effect: 2-3x inference throughput
failure modes: route low when task needs OCR
Decoupled Vision-Language (DvD)
encoder on GPU-A, LLM on GPU-B
stream patch tokens A -> B via NCCL
encoder runs once, LLM runs many steps
bottleneck = max(enc, llm * output_len)
~2x throughput vs co-located
hurts for low-traffic single-request latency