ViT patch-token pipeline — image to transformer input
1. image
H x W x 3 pixels
grid HxW / P^2
2. patchify
N = (H/P)(W/P) patches
each P x P x 3 pixels
flatten -> 3P^2 vector
ViT-B/16 @ 224:
14 x 14 grid = 196 patches
16 x 16 x 3 = 768 pixels
per patch flattened
project
3. linear project
shared W_E (3P^2 x D)
= Conv2d(3, D, k=P, s=P)
each patch -> D-dim
D = 768 (B), 1024 (L)
1152 (SO400m), 1536 (g)
196 patch tokens x D
+ pos
4. + pos/CLS
[CLS] + patches
+ 4 registers (DINOv2)
2D-RoPE in Qwen2-VL
learned pos in ViT-B
seq len
197 (B), 257 (L)
729 (SO400m @384)
downstream: transformer block x L, then pool
transformer blocks
L blocks of attention + MLP
B: L=12, D=768 -> 86M
L: L=24, D=1024 -> 303M
g: L=40, D=1536 -> 1.1B
pretraining
supervised | MAE | DINO
CLIP | SigLIP (2026 pick)
pooling
CLS token (ViT-B, CLIP)
mean patches (DINOv2, SigLIP)
register tokens (sink)
for VLM:
skip pooling entirely
feed all patches to LLM
discard registers
2026 production pick
SigLIP 2 SO400m/14 @ 384
400M params
729 patch tokens per image
4 register tokens
NaFlex native aspect ratio
used by
Qwen2.5-VL, Idefics2
LLaVA-OneVision, InternVL3