DQN: Q-learning + replay + target network
environment
(s, a) -> (r, s')
Atari, GridWorld, whatever
online net Q(.; theta)
acts epsilon-greedy
trained every step
replay buffer D
(s, a, r, s', done)
~1M transitions, uniform sample
target
theta^-
frozen
sync/C
gradient step: one minibatch, TD squared error
for (s, a, r, s', done) in buffer.sample(batch):
y = r if done
y = r + gamma * max_a' Q(s', a'; theta^-) otherwise
loss = (Q(s, a; theta) - y)^2
theta -= lr * d loss / d theta
updated theta acts next step
three tricks, one loss, one neural net: the deep RL era begins