the MDP loop: five objects, one recursion
state s_t
what agent sees
Markov sufficient
policy pi(a | s)
chooses action
a_t ~ pi(. | s_t)
transition P
s_{t+1} ~ P(. | s_t, a_t)
env dynamics
reward r_t
scalar signal
R(s_t, a_t, s_{t+1})
next step: s_{t+1} becomes the new s_t
the Bellman recursion
V^pi(s) = sum_a pi(a|s) · sum_{s',r} P(s',r | s,a) · [ r + gamma · V^pi(s') ]
Q^pi(s,a) = sum_{s',r} P(s',r | s,a) · [ r + gamma · sum_{a'} pi(a'|s') · Q^pi(s',a') ]
iterate to fixed point (DP) · sample (MC) · bootstrap one step (TD)
state + action + transition + reward + discount — every RL algorithm in this phase