six knobs: GPT-2 → open frontier
1 / Norm
LayerNorm — mean + std + scale + bias
RMSNorm — std + scale
2 / Position
learned embedding table
RoPE (+ YaRN / NTK for stretch)
3 / Activation
GELU — one projection
SwiGLU — gate · swish(gate) · up
4 / Attention
MHA — one K,V per head
GQA → MQA → MLA (shrink KV cache)
5 / MLP
dense — every param per token
MoE — top-k of N experts per token
6 / Norm place
post-norm — hard to train deep
pre-norm (stays everywhere)
the diff for each frontier open model
GPT-2 Small
LayerNorm · GELU · learned · MHA · dense · 1k ctx
Llama 3 8B
RMSNorm · SwiGLU · RoPE · GQA 32/8 · dense · 128k
Mixtral 8x7B
RMSNorm · SwiGLU · RoPE · GQA · MoE 8/top-2 · 32k
DeepSeek V3
RMSNorm · SwiGLU · RoPE · MLA · MoE 256/top-8 · 128k · 671B/37B active
same skeleton, six tuned knobs — the next frontier model picks a subset of these