Reinforcement Learning · AI Engineering

MDPs, value methods, policy gradients, PPO, RLHF and GRPO. Formulas, update lines and diagnostics from Phase 9.

PPO training loop

Nodes and connections

Node	Role	Kind
N parallel envs	T steps each
Actor	softmax policy
Rollout buffer	s, a, r, log_pi_old
Critic	v_old per state
GAE	advantages, returns
Clipped surrogate	K epochs, minibatches
Diagnostics	mean KL, clip fraction

From	To	Connection
Actor	N parallel envs	action
N parallel envs	Rollout buffer	transition
Critic	Rollout buffer	value
Rollout buffer	GAE
GAE	Clipped surrogate	advantage
Clipped surrogate	Actor	update
Clipped surrogate	Diagnostics

Sources

RLHF in three stages

SFT fine tune the base model on human demonstrations.
Preference data collect rows of prompt, chosen and rejected.
Reward model train a scalar head on the SFT model with the Bradley-Terry loss.
Policy and reference start the policy from SFT. Freeze a copy as the reference.
Reward RM score minus \beta times the KL to the reference.
PPO update the token level policy against that reward.
Monitor watch mean KL. Past about 5 to 10, drift or reward hacking starts.

Sources

phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md

Value formulas

MDP: the tuple (S, A, P, R, \gamma) with the Markov property.
Return: G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots
Bellman V: V^\pi(s) = \sum_a \pi(a \mid s) \sum_{s', r} P(s', r \mid s, a) [r + \gamma V^\pi(s')]
Value iteration: V(s) \leftarrow \max_a \sum_{s',r} P(s',r \mid s,a) [r + \gamma V(s')]
TD(0): V(s) \leftarrow V(s) + \alpha [r + \gamma V(s') - V(s)]
Q-learning: Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma \max_{a'} Q(s', a') - Q(s, a)]
SARSA: Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma Q(s', a') - Q(s, a)]
DQN loss: (r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta))^2 over replay samples.

Sources

Policy formulas

Policy gradient: \nabla J(\theta) = \mathbb{E}[ \sum_t G_t \cdot \nabla_\theta \log \pi_\theta(a_t \mid s_t) ]
TD advantage: \delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)
GAE: A_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}
PPO ratio: r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)
PPO clip: \mathbb{E}_t [ \min( r_t A_t, \mathop{\mathrm{clip}}(r_t, 1-\epsilon, 1+\epsilon) A_t ) ]
Bradley-Terry: L(\phi) = -\mathbb{E}[ \log \sigma(R_\phi(x, y_+) - R_\phi(x, y_-)) ]
RLHF reward: R_\phi(x, y) - \beta \cdot \mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}})
GRPO advantage: A_i = (r_i - \text{mean}_r) / \text{std}_r over G samples of one prompt.

Sources

Choose a method

Situation	Method
Small tabular MDP, known model	value iteration or policy iteration
Small tabular MDP, samples only	Q-learning
On-policy, safety critical	SARSA or Expected SARSA
High dimensional state, discrete actions	DQN, with Double DQN in production
Continuous control	PPO with Gaussian policy, or SAC and TD3
LLM alignment from preferences	DPO first, RLHF-PPO with KL penalty
LLM math and code reasoning	GRPO with a verifier reward
Two player board games	AlphaZero or MuZero

Sources

Update lines

Task	Code
Return, reverse sweep	`G = r + gamma * G`
Explore with epsilon greedy	`if random() < epsilon: return choice(ACTIONS)`
Q-learning target	`target = r + (gamma * max(Q[s_next].values()) if not done else 0.0)`
SARSA target	`target = r + (gamma * Q[s_next][a_next] if not done else 0.0)`
TD step	`Q[s][a] += alpha * (target - Q[s][a])`
Sync DQN target net	`if steps % sync_every == 0: target = copy(online)`
RLHF reward with KL	`reward = dot(w, bag([token])) - beta * kl(probs, probs_ref)`
GRPO group advantage	`advs = [(r - mean_r) / std_r for r in rewards]`

Sources

PPO clipped update

for rec, adv in zip(mb, adv_norm):advantages normalized per minibatch

x = rec["x"]

probs = softmax(logits(theta, x))current policy

logp = math.log(max(probs[rec["a"]], 1e-12))

ratio = math.exp(logp - rec["log_pi_old"])new over old probability

clipped = (adv > 0 and ratio > 1 + eps) or (adv < 0 and ratio < 1 - eps)

pg_scale = 0.0 if clipped else ratio * advclipped means zero gradient

for action in range(N_ACTIONS):

grad_logpi = (1.0 if action == rec["a"] else 0.0) - probs[action]

for j in range(N_FEAT):

theta[action][j] += lr_a * pg_scale * grad_logpi * x[j]

err = rec["ret"] - value(w, x)critic error

for j in range(N_FEAT):

w[j] += lr_v * err * x[j]critic step

Sources

phases/09-reinforcement-learning/08-ppo/code/main.py

Defaults and numbers

Setting	Default	Note
Discount \gamma	0.99	horizon about 1 / (1 - \gamma), so 100 steps
Tabular \alpha and \epsilon	0.1	Q-learning and SARSA on the 4x4 GridWorld
PPO clip \epsilon	0.2	0.1 is too timid, 0.3 and more is unstable
Value coefficient c_v	0.5	PPO and A2C
Entropy coefficient c_e	0.01	PPO and A2C
GAE \lambda	0.95	0 is TD, 1 is MC
RLHF KL \beta	0.01-0.05	keeps the policy near SFT
GRPO group size G	8	8 to 64 is standard, below 4 is noisy

Sources

Troubleshooting

Symptom	Cause	Fix
Mean KL past 0.1	policy drifts far from the old policy	reduce `K_EPOCHS` or `LR`
Clip fraction near 0	clip never triggers	raise `LR` or `K_EPOCHS`
Clip fraction 0.5 or more	overfit to the rollout	lower `LR` or `K_EPOCHS`
Reward climbs, human eval drops	reward hacking of the RM	stop early, raise \beta, broaden RM data
Responses get longer	RM rewards length	length normalized reward
Q values too high	max bias on noisy Q	Double DQN
DQN diverges	deadly triad	keep the target net and the replay buffer
Policy stops exploring	entropy collapse	entropy bonus with c_e > 0

Sources

Abbreviations

MDP: Markov Decision Process. States, actions, transitions, rewards, discount.
TD: Temporal Difference. Update from a one step bootstrapped target.
DQN: Deep Q-Network. Q-learning with replay, target net and reward clipping.
A2C: Advantage Actor-Critic. Synchronous batched parallel envs.
GAE: Generalized Advantage Estimation. Weighted sum of TD residuals.
PPO: Proximal Policy Optimization. Clipped importance ratio, many epochs.
RLHF: Reinforcement Learning from Human Feedback. SFT, then RM, then PPO.
GRPO: Group Relative Policy Optimization. PPO variant without a critic.

Sources

Questions

On-policy or off-policy?: On-policy learns from its own data. Off-policy learns from another policy's data.
Q-learning or SARSA?: Q-learning bootstraps on the max. SARSA uses the next action it takes and is more cautious.
Why subtract a baseline?: It keeps the gradient unbiased and cuts the variance.
Why the KL in RLHF?: It keeps the policy near the data where the RM was trained.
Why GRPO for reasoning?: No value network, so half the memory. Per prompt normalization fits sparse rewards.

Sources

Tips

Write the five MDP tuples before you write a training loop.
Check a tabular agent against the DP value V^{\ast}.
Decay \epsilon from 1.0 to 0.01 over the first 10% of training.
Normalize advantages per batch to zero mean and unit std.
Compute the ratio as exp(log_new minus log_old) for numerical stability.
Start PPO with K = 10 epochs and minibatches of 64.

Sources

Original source

---
id: reinforcement-learning
title: Reinforcement Learning
code: RL
subtitle: MDPs, value methods, policy gradients, PPO, RLHF and GRPO. Formulas, update lines and diagnostics from Phase 9.
hue: 75
sources: phases/09-reinforcement-learning/01-mdps-states-actions-rewards, phases/09-reinforcement-learning/02-dynamic-programming, phases/09-reinforcement-learning/03-monte-carlo-methods, phases/09-reinforcement-learning/04-q-learning-sarsa, phases/09-reinforcement-learning/05-dqn, phases/09-reinforcement-learning/06-policy-gradients-reinforce, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c, phases/09-reinforcement-learning/08-ppo, phases/09-reinforcement-learning/09-reward-modeling-rlhf, phases/09-reinforcement-learning/12-rl-for-games
source_hash: a115ea75e61f42aa
---

## PPO training loop
type: architecture
from: phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/08-ppo/code/main.py, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md

```arch
node | envs | 1,1 | N parallel envs | T steps each
node | actor | 2,1 | Actor | softmax policy
node | buf | 1,2 | Rollout buffer | s, a, r, log_pi_old
node | critic | 2,2 | Critic | v_old per state
node | gae | 1,3 | GAE | advantages, returns
node | clip | 2,3 | Clipped surrogate | K epochs, minibatches
node | diag | 1,4 | Diagnostics | mean KL, clip fraction
edge | actor | envs | action
edge | envs | buf | transition
edge | critic | buf | value
edge | buf | gae |
edge | gae | clip | advantage
edge | clip | actor | update
edge | clip | diag |
```

## RLHF in three stages
type: flow
from: phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md

1. **SFT**: fine tune the base model on human demonstrations.
2. **Preference data**: collect rows of prompt, chosen and rejected.
3. **Reward model**: train a scalar head on the SFT model with the Bradley-Terry loss.
4. **Policy and reference**: start the policy from SFT. Freeze a copy as the reference.
5. **Reward**: RM score minus $\beta$ times the KL to the reference.
6. **PPO**: update the token level policy against that reward.
7. **Monitor**: watch mean KL. Past about 5 to 10, drift or reward hacking starts.

## Value formulas
type: concepts
from: phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md, phases/09-reinforcement-learning/02-dynamic-programming/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md

- **MDP**: the tuple $(S, A, P, R, \gamma)$ with the Markov property.
- **Return**: $G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \ldots$
- **Bellman V**: $V^\pi(s) = \sum_a \pi(a \mid s) \sum_{s', r} P(s', r \mid s, a) [r + \gamma V^\pi(s')]$
- **Value iteration**: $V(s) \leftarrow \max_a \sum_{s',r} P(s',r \mid s,a) [r + \gamma V(s')]$
- **TD(0)**: $V(s) \leftarrow V(s) + \alpha [r + \gamma V(s') - V(s)]$
- **Q-learning**: $Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma \max_{a'} Q(s', a') - Q(s, a)]$
- **SARSA**: $Q(s, a) \leftarrow Q(s, a) + \alpha [r + \gamma Q(s', a') - Q(s, a)]$
- **DQN loss**: $(r + \gamma \max_{a'} Q(s', a'; \theta^-) - Q(s, a; \theta))^2$ over replay samples.

## Policy formulas
type: concepts
from: phases/09-reinforcement-learning/06-policy-gradients-reinforce/docs/en.md, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

- **Policy gradient**: $\nabla J(\theta) = \mathbb{E}[ \sum_t G_t \cdot \nabla_\theta \log \pi_\theta(a_t \mid s_t) ]$
- **TD advantage**: $\delta_t = r_{t+1} + \gamma V(s_{t+1}) - V(s_t)$
- **GAE**: $A_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}$
- **PPO ratio**: $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_{\text{old}}}(a_t \mid s_t)$
- **PPO clip**: $\mathbb{E}_t [ \min( r_t A_t, \mathop{\mathrm{clip}}(r_t, 1-\epsilon, 1+\epsilon) A_t ) ]$
- **Bradley-Terry**: $L(\phi) = -\mathbb{E}[ \log \sigma(R_\phi(x, y_+) - R_\phi(x, y_-)) ]$
- **RLHF reward**: $R_\phi(x, y) - \beta \cdot \mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}})$
- **GRPO advantage**: $A_i = (r_i - \text{mean}_r) / \text{std}_r$ over $G$ samples of one prompt.

## Choose a method
type: command-table
from: phases/09-reinforcement-learning/02-dynamic-programming/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

| Situation | Method |
|---|---|
| Small tabular MDP, known model | value iteration or policy iteration |
| Small tabular MDP, samples only | Q-learning |
| On-policy, safety critical | SARSA or Expected SARSA |
| High dimensional state, discrete actions | DQN, with Double DQN in production |
| Continuous control | PPO with Gaussian policy, or SAC and TD3 |
| LLM alignment from preferences | DPO first, RLHF-PPO with KL penalty |
| LLM math and code reasoning | GRPO with a verifier reward |
| Two player board games | AlphaZero or MuZero |

## Update lines
type: task-table
from: phases/09-reinforcement-learning/03-monte-carlo-methods/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

| Task | Code |
|---|---|
| Return, reverse sweep | `G = r + gamma * G` |
| Explore with epsilon greedy | `if random() < epsilon: return choice(ACTIONS)` |
| Q-learning target | `target = r + (gamma * max(Q[s_next].values()) if not done else 0.0)` |
| SARSA target | `target = r + (gamma * Q[s_next][a_next] if not done else 0.0)` |
| TD step | `Q[s][a] += alpha * (target - Q[s][a])` |
| Sync DQN target net | `if steps % sync_every == 0: target = copy(online)` |
| RLHF reward with KL | `reward = dot(w, bag([token])) - beta * kl(probs, probs_ref)` |
| GRPO group advantage | `advs = [(r - mean_r) / std_r for r in rewards]` |

## PPO clipped update
type: annotated-code
lang: python
from: phases/09-reinforcement-learning/08-ppo/code/main.py

```python
for rec, adv in zip(mb, adv_norm):                 # advantages normalized per minibatch
    x = rec["x"]
    probs = softmax(logits(theta, x))              # current policy
    logp = math.log(max(probs[rec["a"]], 1e-12))
    ratio = math.exp(logp - rec["log_pi_old"])     # new over old probability
    clipped = (adv > 0 and ratio > 1 + eps) or (adv < 0 and ratio < 1 - eps)
    pg_scale = 0.0 if clipped else ratio * adv     # clipped means zero gradient
    for action in range(N_ACTIONS):
        grad_logpi = (1.0 if action == rec["a"] else 0.0) - probs[action]
        for j in range(N_FEAT):
            theta[action][j] += lr_a * pg_scale * grad_logpi * x[j]
    err = rec["ret"] - value(w, x)                 # critic error
    for j in range(N_FEAT):
        w[j] += lr_v * err * x[j]                  # critic step
```

## Defaults and numbers
type: defaults
from: phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

| Setting | Default | Note |
|---|---|---|
| Discount $\gamma$ | 0.99 | horizon about $1 / (1 - \gamma)$, so 100 steps |
| Tabular $\alpha$ and $\epsilon$ | 0.1 | Q-learning and SARSA on the 4x4 GridWorld |
| PPO clip $\epsilon$ | 0.2 | 0.1 is too timid, 0.3 and more is unstable |
| Value coefficient $c_v$ | 0.5 | PPO and A2C |
| Entropy coefficient $c_e$ | 0.01 | PPO and A2C |
| GAE $\lambda$ | 0.95 | 0 is TD, 1 is MC |
| RLHF KL $\beta$ | 0.01-0.05 | keeps the policy near SFT |
| GRPO group size $G$ | 8 | 8 to 64 is standard, below 4 is noisy |

## Troubleshooting
type: troubleshooting
from: phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md, phases/09-reinforcement-learning/06-policy-gradients-reinforce/docs/en.md, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Mean KL past 0.1 | policy drifts far from the old policy | reduce `K_EPOCHS` or `LR` |
| Clip fraction near 0 | clip never triggers | raise `LR` or `K_EPOCHS` |
| Clip fraction 0.5 or more | overfit to the rollout | lower `LR` or `K_EPOCHS` |
| Reward climbs, human eval drops | reward hacking of the RM | stop early, raise $\beta$, broaden RM data |
| Responses get longer | RM rewards length | length normalized reward |
| Q values too high | max bias on noisy Q | Double DQN |
| DQN diverges | deadly triad | keep the target net and the replay buffer |
| Policy stops exploring | entropy collapse | entropy bonus with $c_e > 0$ |

## Abbreviations
type: glossary
from: phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

- **MDP**: Markov Decision Process. States, actions, transitions, rewards, discount.
- **TD**: Temporal Difference. Update from a one step bootstrapped target.
- **DQN**: Deep Q-Network. Q-learning with replay, target net and reward clipping.
- **A2C**: Advantage Actor-Critic. Synchronous batched parallel envs.
- **GAE**: Generalized Advantage Estimation. Weighted sum of TD residuals.
- **PPO**: Proximal Policy Optimization. Clipped importance ratio, many epochs.
- **RLHF**: Reinforcement Learning from Human Feedback. SFT, then RM, then PPO.
- **GRPO**: Group Relative Policy Optimization. PPO variant without a critic.

## Questions
type: qa
from: phases/09-reinforcement-learning/03-monte-carlo-methods/docs/en.md, phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md, phases/09-reinforcement-learning/06-policy-gradients-reinforce/docs/en.md, phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md, phases/09-reinforcement-learning/12-rl-for-games/docs/en.md

- **On-policy or off-policy?** On-policy learns from its own data. Off-policy learns from another policy's data.
- **Q-learning or SARSA?** Q-learning bootstraps on the max. SARSA uses the next action it takes and is more cautious.
- **Why subtract a baseline?** It keeps the gradient unbiased and cuts the variance.
- **Why the KL in RLHF?** It keeps the policy near the data where the RM was trained.
- **Why GRPO for reasoning?** No value network, so half the memory. Per prompt normalization fits sparse rewards.

## Tips
type: tips
from: phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md, phases/09-reinforcement-learning/02-dynamic-programming/docs/en.md, phases/09-reinforcement-learning/05-dqn/docs/en.md, phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md, phases/09-reinforcement-learning/08-ppo/docs/en.md

- Write the five MDP tuples before you write a training loop.
- Check a tabular agent against the DP value $V^{\ast}$.
- Decay $\epsilon$ from 1.0 to 0.01 over the first 10% of training.
- Normalize advantages per batch to zero mean and unit std.
- Compute the ratio as exp(log_new minus log_old) for numerical stability.
- Start PPO with $K = 10$ epochs and minibatches of 64.