Chameleon — images as discrete tokens in a shared vocabulary
training: one sequence, one loss
1. image
raw RGB pixels
VQ-VAE
2. quantize
codebook K=8192
32x32 = 1024 ints
[4821, 1029, 2891, ...]
one token per patch
reconstruction is lossy
3. shared vocabulary
text ids 0..31999
image ids 32000..40191
<image> / </image> separators
single embedding layer
single next-token cross-entropy loss
inference: mixed-modality generation in one forward pass
prompt: "Draw a cat"
text tokens flow in
model autoregressively emits
<image> code code code ... </image>
then follows with caption text
VQ decoder renders pixels at display time
stability tricks
QK-Norm before dot product
dropout after every residual
final-block skip LN
without these 34B diverges
training recipe is half the paper
trade-offs
+ image gen in same model
+ mixed-modality output free
- tokenizer caps image quality
- expensive inference (VQ decode)
- no LLM reuse
Emu3 + Transfusion extend this path