Transfusion — one transformer, two losses, hybrid attention mask
forward pass, two loss heads
text tokens
discrete BPE vocab
causal attention
teacher forcing
loss: cross-entropy
next-token prediction
vocab-logits head
same as any LLM
shared transformer body
one weight set
block-triangular mask
both modalities in
one forward pass
gradient mixes
text and image objectives
shared body, two heads
image patches
continuous vectors
bidirectional attention
within image block
loss: MSE on velocity
flow-matching diffusion
predict noise -> data
SD3 MMDiT sibling
hybrid attention mask for [T T <image> P P P P </image> T T]
T
T
<I>
P
P
P
P
</I>
T
T
mask rules
1. text-to-text: causal (triangular)
2. patch-to-patch: full bidirectional within image
3. text-to-prior-image: attend fully
4. image-to-prior-text: attend fully
5. no image-to-later-text (causal block)
implemented as a single block-triangular mask