REINFORCE — gradient ascent on expected return
policy gradient theorem
grad J(theta) = E_pi [ sum_t G_t * grad_theta log pi_theta(a_t | s_t) ]
one rollout, one update
roll out tau ~ pi_theta, compute returns G_0, G_1, ..., G_T
for each (s_t, a_t): theta += lr * (G_t - b(s_t)) * grad log pi_theta(a_t | s_t)
sample action
logits = f_theta(s)
p = softmax(logits)
a ~ Categorical(p)
stochastic by design
credit assignment
G_t = sum_{k>=t} g^{k-t} r_{k+1}
reward-to-go only
baseline: G_t - V(s_t)
unbiased, lower variance
apply gradient
grad log pi = e_a - p
theta += lr * adv * grad
+ entropy bonus
on-policy, batched
PPO = REINFORCE + clipped IS ratio · GRPO = REINFORCE + group-relative baseline · DPO = REINFORCE without sampling