Qwen 3 GSPO & DeepSeek GRPO — LLM Reasoning

Modern reasoning models are mostly trained without a critic. GRPO — introduced with DeepSeekMath — replaces the value network with a simpler idea: sample several answers to the same prompt and score them against each other. Qwen's GSPO keeps that advantage but moves the importance ratio from tokens to whole sequences.

The idea

For each prompt $x$, sample a group of $G$ completions $y_1, \dots, y_G$ from the old policy, score each with a reward $R_i$, then normalize inside the group:

Every token of completion $i$ receives the same advantage $\hat{A}_i$, and the per-token ratio is clipped exactly as in PPO. The group mean is the baseline, so no critic is needed — the cost moves from memory and a hard-to-train network to more sampling. The original formulation also subtracts a KL term against a reference model.

GSPO changes the granularity of the ratio. In a long chain of thought a token-level ratio multiplies thousands of factors, so a few outlier tokens dominate the gradient and per-token clipping does not actually bound how far the sequence-level policy moved. GSPO uses a length-normalized sequence ratio:

a geometric mean over tokens, clipped with the same style of surrogate. Two consequences: the trust region matches the unit the reward scores, so whole completions are reinforced or suppressed together, and the ratio no longer grows with sequence length. Whether the per-token loss is averaged over tokens or over sequences also changes the implicit length weighting.

Worked example

A group of four completions scored by a binary verifier: $R = [1, 0, 1, 0]$. Mean $= 0.5$, population standard deviation $= 0.5$, so

The advantages sum to zero by construction: half the group passed, so no completion gets a strong push — the mean absorbs the prompt's difficulty.

Now the ratio granularity. If the average per-token log-ratio is 0.1, the sequence ratio is $e^{0.1} \approx 1.105$ for any length — a 5-token answer and a 2000-token answer land in the same place. The product of per-token ratios for the long answer, by contrast, is roughly $e^{200}$, which is why token-level clipping diverges so badly from sequence-level behavior on long traces.

If all four rewards were identical, every advantage would be zero: a zero-variance group contributes no gradient.

In code

import torch

def grpo_advantages(rewards: torch.Tensor) -> torch.Tensor:      # (G,)
    """Group-relative advantage: the group mean is the baseline, so no critic."""
    return (rewards - rewards.mean()) / (rewards.std(unbiased=False) + 1e-4)

def gspo_ratio(logp, logp_old, mask):                            # (G, T)
    """Length-normalized sequence ratio: a geometric mean over tokens."""
    diff = ((logp - logp_old) * mask).sum(-1) / mask.sum(-1)
    return diff.exp()

mask drops padding tokens; without the division by mask.sum(-1) this is a product again, not a mean.

Check yourself

1. In GRPO, what plays the role of the baseline, and why does that eliminate the value network?
2. Why is a length-normalized sequence ratio better behaved than a product of per-token ratios on a 2000-token reasoning trace?
3. What are the advantages for a group where every completion earns the same reward, and what does that imply for learning?

Key takeaways

• GRPO drops the critic and uses within-group reward normalization as the advantage.
• GSPO moves the importance ratio to the sequence level so the trust region matches the unit the reward scores.
• Both trade a value network for more sampling, and both go silent on zero-variance groups.