DeepSeek-V3 anatomy — 671B total, 37B active
full stack (61 blocks)
embedding (129k x 7168)
3 dense blocks
MLA attention + dense SwiGLU MLP (18432)
58 MoE blocks
MLA attention + MoE MLP
256 experts (2048 hidden each)
+ 1 shared expert (always on)
router: aux-loss-free bias
top-8 routing per token
active per token:
8 routed + 1 shared = 9 experts
KV cache: 512-dim MLA latent
per token per layer
1 MTP module
final RMSNorm + LM head (tied)
total 671B · active 37B · ratio 5.5%
four DeepSeek innovations
1 / MLA — Multi-Head Latent Attention
compress K and V into shared 512-dim latent c^{KV};
KV cache stores 512 floats/token/layer instead of 2 * heads * head_dim.
at 128k: 7.6 GB vs 30.5 GB for GQA(8/128). 4x savings.
2 / MTP — Multi-Token Prediction
D=1 module predicts t+2 from h^(1) + E(t+1).
denser training signal at 2.1% param overhead (14B of 671B).
at inference: 80%+ acceptance as speculative-decoding draft.
3 / aux-loss-free routing
per-expert bias terms adjusted during training
to balance load — no auxiliary loss term needed.
same effect, cleaner objective.
4 / DualPipe training
bidirectional pipeline overlap on 2k H800 GPUs.
hides cross-node all-to-all inside compute windows.
recovers ~245k GPU-hours vs 1F1B at V3's scale.
what this unlocks
· 128k context with a manageable KV cache (MLA)
· larger capacity for same active FLOP budget (256 experts, top-8)
· 1.8x decode throughput free (MTP as spec-decode draft)
· 95%+ GPU utilization at 2k-rank training (DualPipe)