PPO clipped surrogate: flat roof, flat floor
L^{CLIP}(theta) = E[ min( r A, clip(r, 1-eps, 1+eps) * A ) ]
r = pi_theta(a|s) / pi_old(a|s) typical eps = 0.2
advantage A > 0 (good action)
r
L
r = 1+eps
r = 1
clip: flat roof
gradient flows
advantage A < 0 (bad action)
r
L
r = 1-eps
r = 1
clip: flat floor
gradient flows
if the ratio drifts past the clip in the "useful" direction, gradient goes to zero — no unbounded updates