scaling laws — parameters, tokens, compute, and what changed in 2026
loss vs training compute
log10 compute (FLOPs)
cross-entropy loss
18
20
22
24
26
3.5
2.8
2.2
1.9
1.7
Kaplan 2020 (under-trained)
Chinchilla 2022 (D/N ≈ 20)
E (irreducible)
GPT-3
Chinchilla 70B
Llama 3 70B
compute = 6 × N × D
N
parameters (capacity)
D
training tokens (usage)
trade at fixed C
min L = A/N^α + B/D^β + E
α ≈ 0.34, β ≈ 0.28, E ≈ 1.69
the 2026 twist
— Chinchilla-optimal: minimize TRAINING loss per FLOP
— inference-optimal: bigger D, smaller N (over-train)
— Llama 3 8B: D/N ≈ 1875 (15T tokens, 8B params)
— data quality (Phi-style filtering) >2× effective FLOPs
— MoE decouples total params from active FLOPs
— Muon optimizer: 2× training-FLOP efficiency vs AdamW
— "emergence" is often a scoring artifact; loss is smooth
same power-law shape, different absolute constants. still the map frontier labs plan against.