Open VLM Families: LLaVA, Qwen-VL, InternVL · AI Engineering

LLaVA, LLaVA-OneVision, Qwen-VL and InternVL3 side by side: connector, resolution, video, tokens and training.

Families side by side

Factor	LLaVA-NeXT	LLaVA-OneVision	Qwen-VL family	InternVL3
Vision to LLM	2-layer MLP, GELU	Same MLP projector	Qwen2-VL: 2-layer MLP on merged patches	Native, no align stage
Recipe	Post-hoc on a text LLM	Post-hoc on a text LLM	Post-hoc on a text LLM	Native multimodal pretrain
High resolution	AnyRes: 336 tiles plus thumbnail	AnyRes-9 or AnyRes-4, 384 tiles	Qwen2-VL: any H and W divisible by 28	ViR picks low, medium or high
Video	Per-frame concat	32 frames, 196 tokens each	Dynamic FPS, absolute-time M-RoPE	5% video-text in pretrain
Visual tokens	576 base, 2880 AnyRes	6000-9000 per sample, balanced	Scales with input	256, 576 or 2048+ via ViR
Training	Align, then instruct	Stages SI, OV, TT	Qwen3-VL: stages stress OCR, grounding, agents	One pretrain stage, then instruct
Swap the LLM	Retrain the projector	Retrain the projector	Retrain the adapter	Not possible later

Sources

LLaVA forward pass

Nodes and connections

Node	Role	Kind
Image	336x336	client
CLIP ViT-L/14 @ 336	576 patches, dim 1024	model
2-layer MLP	1024 → 4096 → 4096, GELU	process
Prompt	system + <image> + user	client
Replace <image>	576 projected tokens	process
LLM	Vicuna-13B, later Llama-3.1-8B	model

From	To	Connection
Image	CLIP ViT-L/14 @ 336
CLIP ViT-L/14 @ 336	2-layer MLP	patches
2-layer MLP	Replace <image>	576 tokens
Prompt	Replace <image>
Replace <image>	LLM	full sequence

Sources

phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md

Qwen-VL lineage

Qwen-VL (2023) 448 res, grounding boxes as text, Q-Former with 256 queries.
Qwen2-VL (2024) M-RoPE, native dynamic resolution, MLP projector.
Qwen2.5-VL (2025) dynamic FPS, absolute-time M-RoPE, window attention in the ViT.
Qwen2.5-VL agents JSON tool calls. ScreenSpot goes from 55% to 84%.
Qwen3-VL (2025) Qwen3 base, thinking mode, more OCR data. ViT and M-RoPE stay.

Sources

Use each family when

LLaVA you need a cheap, reproducible recipe. Stage 2 takes ~20 hours on 8xA100.
LLaVA-OneVision one model serves single image, multi-image and video at a predictable cost.
Qwen2.5-VL video with timestamps, GUI agents with JSON output, min_pixels and max_pixels caps.
InternVL3 text skills must not regress and you can pay for native pretraining.
Post-hoc recipe you want to swap in a new LLM later. It stays cheaper for most projects.
AnyRes tiling the encoder is frozen at 336x336, but you need detail.

Sources

Toy code by lesson

Task	Code
Project ViT patches	`MLPProjector(PATCH_DIM, HIDDEN_DIM, LLM_DIM)`
Build a LLaVA prompt	`build_llava_prompt(system, user, 576)`
Pack a mixed batch	`pack_batch(images, patch)`
Dense block-diagonal mask	`build_dense_mask(pack)`
AnyRes token cost	`anyres_cost(img, tile=336, thumb=336)`
Square-resize cost	`square_cost(img, side=336, patch=14)`
OneVision video plan	`plan_video(budget)`
ViR average tokens	`vir_sim(tiers)`

Sources

M-RoPE, dynamic FPS, tool calls

cfg = MRoPEConfig(hidden=48, temporal_dim=16, height_dim=16, width_dim=16)

mrope_angles(cfg, 0, 5, 7)image patch: t=0, h=row, w=col

mrope_angles(cfg, 3, 5, 7)video frame t=3

mrope_rotate(cfg, vec, 3, 5, 7) # rotates each band by its axis

plan = VideoPlan(duration_s=600.0, tokens_per_frame=81,

budget=32768, motion="low")

plan.fps()first candidate <= fps_max

plan.frame_times()one timestamp per frame

plan.total_tokens()frames times tokens_per_frame

parse_tool_call(raw)json.loads, then slice { to }

Sources

phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/code/main.py

Token math

LLaVA base: 576 tokens per image. At 2048 context, 1472 tokens stay for text.
AnyRes 2x2: 4 * 576 + 576 = 2880 visual tokens.
AnyRes 4x4: a 1344x1344 image gives 9216 + 576, about 9800 tokens.
Qwen2-VL native: 1120x672 gives 40x24 merged patches, 960 tokens.
OneVision image: AnyRes-9 with 2x2 pool, 9 * 182 + 182 = 1820 tokens.
OneVision video: 32 frames with 2x2 pool, 32 * 196 = 6272 tokens.
Dynamic FPS cap: fps_{max} = B / (T \cdot \text{tokens per frame})
Qwen2.5-VL video: 60 s at 4 FPS with 81 tokens per frame is 19440 tokens.

Sources

Training recipes

Step	LLaVA	OneVision	InternVL3
First	Train MLP only, 558k captions	SI: single image, AnyRes	Native pretrain
Second	MLP + LLM, 158k GPT-4 turns	OV: 0.5 / 0.3 / 0.2 mix	Instruction tuning
Third	None	TT: target task mix	None
Frozen	ViT and LLM in stage 1	No weight reset	No freezing stage
Data	LAION-CC-SBU, LLaVA-Instruct	LLaVA-NeXT data plus own	40/35/20/5 text, interleaved, caption, video

Sources

Defaults in the lessons

Setting	Default	Note
`anyres_cost` tile, thumb	336	576 tokens per tile at patch 14
`square_cost` side	336	Fixed 576 tokens per image
LLaVA prompt image_tokens	576	Replaces the image placeholder
`MRoPEConfig` base	10000.0	RoPE frequency base per band
Qwen2-VL video sampling	1-2 FPS	Accepts any frame count
OneVision toy planner budget	4096	Visual tokens per sample in the lesson code
Dynamic-FPS demo budget	32768	81 tokens per frame after 3x pool
`dvd_throughput` llm_tokens	128	Output tokens per request

Sources

Troubleshooting

Symptom	Cause	Fix
Squished text, lost chart labels	Square resize to 336	Native aspect ratio with a max-pixels cap
Large image fills the context	AnyRes 4x4 is about 9800 tokens	Native pack or M-RoPE, no tile overhead
Patches attend across images	No block-diagonal mask	Mask from cumulative offsets, or use the `cu_seqlens` path
Weak image skills after training	Video-first or multi-image-first order	Train SI, then OV, then TT
GSM8K drops 5-10 points	Post-hoc alignment debt	Native pretrain with a 40% text floor
Long video runs out of frequency range	Fixed position scale	MRoPE-v2 scaling with max input size

Sources

Abbreviations

M-RoPE: Multimodal RoPE. Time, height and width bands in the hidden dim.
V2PE: variable visual position encoding. Learned band split per modality.
NaFlex: SigLIP 2 native-flex mode. One checkpoint serves 256, 729 or 1024 tokens.
ViR: Visual Resolution Router. Picks the minimum resolution before encoding.
DvD: Decoupled Vision-Language deployment. Encoder and LLM on separate GPUs.
SI, OV, TT: OneVision stages. Single-image, OneVision mix, task transfer.

Sources

Questions

Why did MLP beat Q-Former?: LM loss only and easy LLM swap. Contexts grew to 32k-128k+, so 576 tokens became cheap.
OneVision or Qwen2.5-VL budget?: OneVision fixes the budget and scales the pooling. Qwen2.5-VL budget scales with input.
Why does M-RoPE need no position table?: Each token rotates by its (t, h, w) position. Images use t = 0.
Why not always pretrain native?: Compute is like a text LLM. Interleaved data is rare. The LLM cannot change later.
What does DvD give?: For an 8B LLM with a 400M encoder, about 2x per-node throughput.

Sources

Original source

---
id: vlm-model-families
title: Open VLM Families: LLaVA, Qwen-VL, InternVL
code: OV
subtitle: LLaVA, LLaVA-OneVision, Qwen-VL and InternVL3 side by side: connector, resolution, video, tokens and training.
hue: 265
sources: phases/12-multimodal-ai/05-llava-visual-instruction-tuning, phases/12-multimodal-ai/06-any-resolution-patch-n-pack, phases/12-multimodal-ai/08-llava-onevision-single-multi-video, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps, phases/12-multimodal-ai/10-internvl3-native-multimodal
source_hash: 854d13f0d4ee45ed
---

## Families side by side
type: comparison
span: 2
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/code/main.py

| Factor | LLaVA-NeXT | LLaVA-OneVision | Qwen-VL family | InternVL3 |
|---|---|---|---|---|
| Vision to LLM | 2-layer MLP, GELU | Same MLP projector | Qwen2-VL: 2-layer MLP on merged patches | Native, no align stage |
| Recipe | Post-hoc on a text LLM | Post-hoc on a text LLM | Post-hoc on a text LLM | Native multimodal pretrain |
| High resolution | AnyRes: 336 tiles plus thumbnail | AnyRes-9 or AnyRes-4, 384 tiles | Qwen2-VL: any H and W divisible by 28 | ViR picks low, medium or high |
| Video | Per-frame concat | 32 frames, 196 tokens each | Dynamic FPS, absolute-time M-RoPE | 5% video-text in pretrain |
| Visual tokens | 576 base, 2880 AnyRes | 6000-9000 per sample, balanced | Scales with input | 256, 576 or 2048+ via ViR |
| Training | Align, then instruct | Stages SI, OV, TT | Qwen3-VL: stages stress OCR, grounding, agents | One pretrain stage, then instruct |
| Swap the LLM | Retrain the projector | Retrain the projector | Retrain the adapter | Not possible later |

## LLaVA forward pass
type: architecture
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md

```arch
node | img | 1,1 | Image | 336x336 | client
node | vit | 2,1 | CLIP ViT-L/14 @ 336 | 576 patches, dim 1024 | model
node | mlp | 3,1 | 2-layer MLP | 1024 → 4096 → 4096, GELU | process
node | prompt | 1,2 | Prompt | system + <image> + user | client
node | splice | 2,2 | Replace <image> | 576 projected tokens | process
node | llm | 3,2 | LLM | Vicuna-13B, later Llama-3.1-8B | model
edge | img | vit |
edge | vit | mlp | patches
edge | mlp | splice | 576 tokens
edge | prompt | splice |
edge | splice | llm | full sequence
```

## Qwen-VL lineage
type: flow
from: phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/code/main.py

1. **Qwen-VL (2023)**: 448 res, grounding boxes as text, Q-Former with 256 queries.
2. **Qwen2-VL (2024)**: M-RoPE, native dynamic resolution, MLP projector.
3. **Qwen2.5-VL (2025)**: dynamic FPS, absolute-time M-RoPE, window attention in the ViT.
4. **Qwen2.5-VL agents**: JSON tool calls. ScreenSpot goes from 55% to 84%.
5. **Qwen3-VL (2025)**: Qwen3 base, thinking mode, more OCR data. ViT and M-RoPE stay.

## Use each family when
type: tips
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, phases/12-multimodal-ai/06-any-resolution-patch-n-pack/docs/en.md, phases/12-multimodal-ai/06-any-resolution-patch-n-pack/code/main.py, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md

- **LLaVA**: you need a cheap, reproducible recipe. Stage 2 takes ~20 hours on 8xA100.
- **LLaVA-OneVision**: one model serves single image, multi-image and video at a predictable cost.
- **Qwen2.5-VL**: video with timestamps, GUI agents with JSON output, `min_pixels` and `max_pixels` caps.
- **InternVL3**: text skills must not regress and you can pay for native pretraining.
- **Post-hoc recipe**: you want to swap in a new LLM later. It stays cheaper for most projects.
- **AnyRes tiling**: the encoder is frozen at 336x336, but you need detail.

## Toy code by lesson
type: task-table
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py, phases/12-multimodal-ai/06-any-resolution-patch-n-pack/code/main.py, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/code/main.py, phases/12-multimodal-ai/10-internvl3-native-multimodal/code/main.py

| Task | Code |
|---|---|
| Project ViT patches | `MLPProjector(PATCH_DIM, HIDDEN_DIM, LLM_DIM)` |
| Build a LLaVA prompt | `build_llava_prompt(system, user, 576)` |
| Pack a mixed batch | `pack_batch(images, patch)` |
| Dense block-diagonal mask | `build_dense_mask(pack)` |
| AnyRes token cost | `anyres_cost(img, tile=336, thumb=336)` |
| Square-resize cost | `square_cost(img, side=336, patch=14)` |
| OneVision video plan | `plan_video(budget)` |
| ViR average tokens | `vir_sim(tiers)` |

## M-RoPE, dynamic FPS, tool calls
type: annotated-code
lang: python
from: phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/code/main.py

```python
cfg = MRoPEConfig(hidden=48, temporal_dim=16, height_dim=16, width_dim=16)
mrope_angles(cfg, 0, 5, 7)      # image patch: t=0, h=row, w=col
mrope_angles(cfg, 3, 5, 7)      # video frame t=3
mrope_rotate(cfg, vec, 3, 5, 7) # rotates each band by its axis
plan = VideoPlan(duration_s=600.0, tokens_per_frame=81,
                 budget=32768, motion="low")
plan.fps()                      # first candidate <= fps_max
plan.frame_times()              # one timestamp per frame
plan.total_tokens()             # frames times tokens_per_frame
parse_tool_call(raw)            # json.loads, then slice { to }
```

## Token math
type: concepts
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, phases/12-multimodal-ai/06-any-resolution-patch-n-pack/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md

- **LLaVA base**: 576 tokens per image. At 2048 context, 1472 tokens stay for text.
- **AnyRes 2x2**: 4 * 576 + 576 = 2880 visual tokens.
- **AnyRes 4x4**: a 1344x1344 image gives 9216 + 576, about 9800 tokens.
- **Qwen2-VL native**: 1120x672 gives 40x24 merged patches, 960 tokens.
- **OneVision image**: AnyRes-9 with 2x2 pool, 9 * 182 + 182 = 1820 tokens.
- **OneVision video**: 32 frames with 2x2 pool, 32 * 196 = 6272 tokens.
- **Dynamic FPS cap**: $fps_{max} = B / (T \cdot \text{tokens per frame})$
- **Qwen2.5-VL video**: 60 s at 4 FPS with 81 tokens per frame is 19440 tokens.

## Training recipes
type: comparison
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/code/main.py, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md

| Step | LLaVA | OneVision | InternVL3 |
|---|---|---|---|
| First | Train MLP only, 558k captions | SI: single image, AnyRes | Native pretrain |
| Second | MLP + LLM, 158k GPT-4 turns | OV: 0.5 / 0.3 / 0.2 mix | Instruction tuning |
| Third | None | TT: target task mix | None |
| Frozen | ViT and LLM in stage 1 | No weight reset | No freezing stage |
| Data | LAION-CC-SBU, LLaVA-Instruct | LLaVA-NeXT data plus own | 40/35/20/5 text, interleaved, caption, video |

## Defaults in the lessons
type: defaults
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/code/main.py, phases/12-multimodal-ai/06-any-resolution-patch-n-pack/code/main.py, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/code/main.py, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/code/main.py, phases/12-multimodal-ai/10-internvl3-native-multimodal/code/main.py

| Setting | Default | Note |
|---|---|---|
| `anyres_cost` tile, thumb | 336 | 576 tokens per tile at patch 14 |
| `square_cost` side | 336 | Fixed 576 tokens per image |
| LLaVA prompt image_tokens | 576 | Replaces the image placeholder |
| `MRoPEConfig` base | 10000.0 | RoPE frequency base per band |
| Qwen2-VL video sampling | 1-2 FPS | Accepts any frame count |
| OneVision toy planner budget | 4096 | Visual tokens per sample in the lesson code |
| Dynamic-FPS demo budget | 32768 | 81 tokens per frame after 3x pool |
| `dvd_throughput` llm_tokens | 128 | Output tokens per request |

## Troubleshooting
type: troubleshooting
from: phases/12-multimodal-ai/06-any-resolution-patch-n-pack/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Squished text, lost chart labels | Square resize to 336 | Native aspect ratio with a max-pixels cap |
| Large image fills the context | AnyRes 4x4 is about 9800 tokens | Native pack or M-RoPE, no tile overhead |
| Patches attend across images | No block-diagonal mask | Mask from cumulative offsets, or use the `cu_seqlens` path |
| Weak image skills after training | Video-first or multi-image-first order | Train SI, then OV, then TT |
| GSM8K drops 5-10 points | Post-hoc alignment debt | Native pretrain with a 40% text floor |
| Long video runs out of frequency range | Fixed position scale | MRoPE-v2 scaling with max input size |

## Abbreviations
type: glossary
from: phases/12-multimodal-ai/06-any-resolution-patch-n-pack/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md

- **M-RoPE**: Multimodal RoPE. Time, height and width bands in the hidden dim.
- **V2PE**: variable visual position encoding. Learned band split per modality.
- **NaFlex**: SigLIP 2 native-flex mode. One checkpoint serves 256, 729 or 1024 tokens.
- **ViR**: Visual Resolution Router. Picks the minimum resolution before encoding.
- **DvD**: Decoupled Vision-Language deployment. Encoder and LLM on separate GPUs.
- **SI, OV, TT**: OneVision stages. Single-image, OneVision mix, task transfer.

## Questions
type: qa
from: phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md, phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md, phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md, phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md

- **Why did MLP beat Q-Former?** LM loss only and easy LLM swap. Contexts grew to 32k-128k+, so 576 tokens became cheap.
- **OneVision or Qwen2.5-VL budget?** OneVision fixes the budget and scales the pooling. Qwen2.5-VL budget scales with input.
- **Why does M-RoPE need no position table?** Each token rotates by its (t, h, w) position. Images use t = 0.
- **Why not always pretrain native?** Compute is like a text LLM. Interleaved data is rare. The LLM cannot change later.
- **What does DvD give?** For an 8B LLM with a 400M encoder, about 2x per-node throughput.