PPO, LLM Reasoning, Importance Ratio, Advantage

PPO is the bridge between the RL fundamentals and how today's reasoning models are trained. It fixes the biggest practical weakness of REINFORCE — that one rollout licenses one gradient step — by clipping how far the policy is allowed to move per update. Replace the environment with a text prompt and the action with a token, and the same clipped objective becomes RLHF and verifier-based RL.

The idea

Importance sampling lets data collected by an older policy $\pi_{\theta_{old}}$ be reused to estimate the objective of the current one. PPO's clipped surrogate is

The min makes the objective a pessimistic lower bound:

• With $A_t > 0$, the gain stops growing once the ratio passes $1+\epsilon$ — no reward for over-exploiting a lucky batch.
• With $A_t < 0$, the penalty stops growing once the ratio falls below $1-\epsilon$ — no destruction of the policy from one bad batch.

The advantage comes from a learned critic. GAE blends one-step TD errors over a rollout, $\hat{A}_t = \sum_l (\gamma\lambda)^l \delta_{t+l}$, trading bias for variance; the batch is usually normalized to zero mean and unit standard deviation. Typical settings: $\epsilon$ of 0.1–0.2, 2–10 epochs per rollout, and an early stop on KL divergence.

The LLM correspondence: state is the prompt plus tokens generated so far, action is the next token, reward is one scalar from a reward model or verifier at the end, and a per-token KL penalty to the frozen reference model keeps generation from drifting. The cost is memory: policy, reference, reward model, and critic all resident at once.

Worked example

With $\epsilon = 0.2$ the clip range is $[0.8, 1.2]$.

For a positive advantage $A_t = +2$:

• ratio 1.0: $\min(2.0,\ 2.0) = 2.0$ — no clipping.
• ratio 1.5: $\min(3.0,\ 2.4) = 2.4$ — clipped, and the gradient is zero from here on.
• ratio 0.5: $\min(1.0,\ 1.6) = 1.0$ — unclipped; raising the probability of a good action is always allowed.

For a negative advantage $A_t = -2$:

• ratio 0.5: $\min(-1.0,\ -1.6) = -1.6$ — clipped; the penalty does not grow past the point where the ratio undershot.
• ratio 1.5: $\min(-3.0,\ -2.4) = -3.0$ — active; the objective pushes the ratio back down.

The asymmetry is the whole idea: the clip removes the incentive to keep moving in the direction that already helped, and does nothing else.

In code

import torch

def ppo_loss(logp, logp_old, advantages, eps=0.2):
    ratio = (logp - logp_old).exp()
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * advantages
    return -torch.min(unclipped, clipped).mean()   # negate: the optimizer descends

logp is per token and advantages is broadcast over the sequence. The - is not a detail; forget it and the policy is pushed the wrong way.

Check yourself

1. Why does the importance ratio make it valid to take several gradient steps on one rollout?
2. With $A_t = +2$ and $\epsilon = 0.2$, what happens to the objective once the ratio reaches 1.5?
3. In the LLM setting, what plays the role of the state, the action, and the reward?

Key takeaways

• PPO is REINFORCE plus importance sampling plus a pessimism clip that bounds each update.
• The clip, not the learning rate, defines the trust region; the critic supplies the advantages.
• For LLMs: state is context, action is a token, reward arrives at the end, and a KL leash holds the policy near the reference model.