DRPO: A Smoother Trust Region for Reinforcement Learning on LLMs
Researchers from Tencent Hunyuan, UIUC and NUS propose DRPO, which limits how far each token's probability may move during RL training and pulls it back when it overshoots.
TL;DR
- Researchers at Tencent Hunyuan, UIUC and NUS propose DRPO, which limits how far each token's probability moves during RL training using a smooth penalty [^1].
- In math-reasoning tests across six model and precision settings, it matched or beat the best baseline, while ratio-based methods often collapsed at FP8 [^1].
Background
Reinforcement learning is now a central step in post-training LLMs: the model writes a response, a reward model or rule-based checker scores it, and the model is updated [^1]. The responses usually come from a slightly older or numerically different copy of the model [^1]. A trust region limits how far each update may drift from that copy. PPO, much simpler to implement than its predecessor TRPO [^2], enforces it by clipping each token's probability ratio [^1].
What happened
The paper, by researchers at Tencent Hunyuan, UIUC and NUS, starts from a known weakness of that ratio, which compares a token's probability under the model being trained with its probability when the response was generated [^1]. LLM vocabularies are large and long-tailed. A small increase on a rare token can produce a huge ratio while moving almost no probability, and a moderate ratio change on a common token can move a lot, so a fixed ratio window over-constrains rare tokens and under-constrains common ones [^1]. The tail matters: in Qwen3-30B-A3B-Base, tokens sampled with a probability of 0.01 or less made up 7.8 percent of all sampled tokens [^1]. An earlier method, DPPO, measures the absolute probability shift of the sampled token instead and switches off its gradient once it crosses a threshold in the wrong direction. That hard mask, the authors argue, gives no signal to move the token back and causes abrupt changes near the threshold [^1].
DRPO, short for Divergence Regularized Policy Optimization, keeps DPPO's boundary but replaces the mask with a smooth quadratic penalty on that absolute shift, borrowing the construction SPO used to smooth out PPO's clipping [^1]. Each token's gradient is multiplied by a continuous weight. If an update pushes the token's probability further from where it started, the weight shrinks to zero at the boundary and turns negative beyond it, pulling the token back; if the update moves it back, the weight grows [^1]. Because the shift is a difference between two probabilities, the weight stays bounded, while SPO's ratio-based weight can grow without limit for rare tokens [^1].
The team fine-tuned Qwen3-4B-Base, Qwen3-30B-A3B-Base and Qwen3.5-35B-A3B-Base on about 13K math problems from the DAPO dataset with rule-based verification, plus DeepSeek-R1-Distill-Qwen-1.5B on a sanity set of 1,460 questions, and scored them on AIME 2024 and 2025 [^1]. For the 30B mixture-of-experts model, they added runs with FP8 precision for rollouts only and for both training and rollouts, which can widen the numerical gap between the two engines [^1]. Across all six settings, DRPO matched or exceeded the best baseline accuracy. GRPO and SPO were generally unstable and often collapsed in the low-precision runs, and DPPO trained stably on Qwen3-30B-A3B-Base but converged more slowly to a lower final accuracy [^1].
Why it matters
Dropping the trust region entirely sometimes worked well, but not reliably: it lost accuracy in three of the six settings, falling from 0.25 to 0.17 on Qwen3-4B-Base [^1]. The authors read this as support for the view that a trust region is still needed and that its form is crucial [^1]. Their ablations push the point further. Standard KL or TV penalties underperformed DRPO, which the authors attribute to gradients that reintroduce ratio-based behavior. Removing the weighting by each token's advantage, the measure of how much better a response scored than expected, caused performance drops and instability [^1]. Their conclusion is that the gradient a penalty induces matters more than the divergence it nominally measures [^1].
For teams running RL on LLMs, the tuning result may matter as much as the accuracy. DPPO needed its threshold tuned per setting, with 0.15 best on Qwen3-30B-A3B-Base and 0.6 in the FP8 end-to-end run, while DRPO used a single value of 12.5 and lost only a little when it was cut to 2.5 [^1]. The evidence has limits. All main runs used math problems with rule-checked answers, and the Qwen3.5 model got only 110 training steps, against 300 for Qwen3-30B-A3B-Base, because the training backend lacked efficient support for it at the time [^1].
Practical example
Say Priya runs post-training at a small lab and fine-tunes Qwen3-30B-A3B-Base on math problems with an automatic answer checker. To speed things up, Priya moves both rollouts and training to FP8. With a ratio-based method, the run collapses before reaching reasonable accuracy, as GRPO and SPO often did in the paper's low-precision runs [^1].
The ratio explains part of it. A rare token can show an enormous ratio after a tiny change in probability: under PPO-style clipping it gets cut off, and under SPO's smooth penalty its weight can dominate the gradient. Meanwhile a common token that loses real probability mass passes the check [^1].
Priya switches to DPPO, which measures the absolute probability shift instead. The run stays stable, but now the threshold needs tuning for each setup: in the paper, 0.15 worked best for one configuration and 0.6 for the FP8 end-to-end one [^1]. Tokens that cross the line simply stop receiving gradient, with nothing pulling them back [^1].
With DRPO, a token drifting away from its starting probability gets a weaker and weaker push as it nears the boundary, and a push in the opposite direction once it passes it [^1]. Priya can keep one setting, 12.5, which the authors used in all their main runs [^1]. What Priya cannot assume is that the gain carries over to coding or chat tasks: the paper's main runs only measured math [^1].
Related gear
We recommend this book because it explains reinforcement learning and policy-gradient methods from the ground up, the family of algorithms that PPO, GRPO and DRPO belong to.