actor-critic: policy + value, trained together
shared state features x = phi(s)
MLP trunk, CNN on pixels, transformer on tokens — same input, two heads
actor pi_theta(a | s)
logits = f_theta(x)
p = softmax(logits)
loss: -A * log pi(a|s)
+ c_e * H(pi(.|s))
policy gradient, on-policy
critic V_phi(s)
v_hat = w . x
target = G_t (MC)
or r + g V(s') (TD)
loss: c_v * (v_hat - target)^2
MSE regression to return
advantage estimator A_t
A_t = delta_t + (g * lam) * A_{t+1} delta_t = r_{t+1} + g V(s_{t+1}) - V(s_t)
GAE(lam): lam=0 pure TD, lam=1 pure MC, lam=0.95 the 2026 default
subtract V from return, normalize, multiply by grad log pi — lower variance, same gradient