sequential multi-token prediction — DeepSeek-V3 style
main model backbone
h_i^(0) · L_main
shared embedding E
used by main + every MTP depth
MTP module k=1
combine: M_1 * [RMSNorm(h^0); RMSNorm(E(t_{i+1}))]
transformer block T_1 (attn + MLP)
h_i^(1)
shared LM head -> predict t_{i+2}
L_1 = CE(logits, t_{i+2})
MTP module k=2
combine: M_2 * [RMSNorm(h^1); RMSNorm(E(t_{i+2}))]
transformer block T_2 (attn + MLP)
h_i^(2)
shared LM head -> predict t_{i+3}
L_2 = CE(logits, t_{i+3})
h^(1)
shared output head Out
reuses main model's LM head — no extra params
joint loss
L_total = L_main + (lambda / D) * sum_{k=1..D} L_k
at inference: MTP modules become a speculative-decoding draft
module k=1
draft t_{i+2} given h^(0) + E(t_{i+1}) · DeepSeek-V3 reports 80%+ acceptance
module k=2
draft t_{i+3} given h^(1) + E(t_{i+2}) · chains across depths
verifier
main model accepts/rejects via Leviathan rule (Phase 10 · 15)
speedup
~1.8x throughput in generation — paper-reported measured value