mixture of experts — total params grow, active params stay flat
token x
router (top-k of E experts)
E_1
skip
E_2
g=0.52
E_3
skip
E_4
skip
E_5
g=0.30
E_6
skip
E_7
skip
E_8
g=0.18
Σ g_k · E_k(x)
+ shared expert (always on)
auxiliary-loss-free balance
each expert has a bias b_e
selection uses score + bias
gate weight uses score only
bias nudged ±γ based on usage — no extra loss
DeepSeek-V3: 1 shared + 256 routed, top-8. Total 671B, active 37B per token.