self-play + search + policy improvement — one recipe, three eras
AlphaZero (2017)
env: chess / shogi / Go
rules: known
policy-value net (p, v)
MCTS with PUCT
target: pi_MCTS, z_outcome
L = (v-z)^2 - pi * log p
tabula rasa self-play
Go, chess, shogi
requires: perfect sim
discrete actions
zero-sum
MuZero (2019)
env: anything
rules: learned (g)
encoder h, dynamics g
MCTS in latent space
target: pi_MCTS, z, reward
L = sum over horizon
no sim required
Atari, Go, chess, shogi
same loop, learned model
higher compute cost
stochastic extensions exist
GRPO (2024-2025)
env: LLM reasoning
action: next token
verifier: tests/math check
group sampling (G rollouts)
A_i = (r_i - mean_r) / std_r
L = -A * log pi + beta KL
no critic, no reward model
math, code, reasoning
DeepSeek-R1 recipe
group-relative baseline
same spirit as AlphaZero
pattern: sample many trajectories, rank them, train the policy to increase the good ones' probability