Q-learning vs SARSA — one line of code apart
the transition
(s, a) --env.step--> (r, s') (one step, online, no episode boundary needed)
Q-learning (off-policy)
target = r + gamma * max_{a'} Q(s', a')
Q(s, a) += alpha * (target - Q(s, a))
assumes greedy policy from s' onward
learns Q* even while exploring
max bias: overestimates noisy Q
deep version: DQN
SARSA (on-policy)
a' = epsilon_greedy(Q, s')
target = r + gamma * Q(s', a')
Q(s, a) += alpha * (target - Q(s, a))
uses the ACTION actually taken
learns Q^pi for current epsilon-greedy
conservative near cliffs / danger
same shape: target - Q(s,a), scaled by alpha
the ONE symbol difference: max vs sampled a' — off-policy vs on-policy
everything in Phase 9 after this lesson is deep-function-approximated variants of these two updates