Progress Advantage Scores Agent Steps Without a Reward Model
Researchers show that comparing an RL-trained model with its reference checkpoint yields a step-level score for agent runs, with no labels and no separate reward model to train.
TL;DR
- Researchers show that the log-probability ratio between an RL-trained model and its earlier checkpoint scores an agent's steps, with no reward model to train [^1].
- Used to pick the best of eight agent attempts, it averaged 38.8 and 62.1 percent success on two models, ahead of trained reward models [^1].
Background
When an agent works through a long task, one score at the end says little about which step went wrong. Process reward models score each step instead, but the authors say building them for agents is very hard [^1]. Tasks span many steps, actions such as sending an email or deleting a file can't be undone, and both human step labels and Monte Carlo estimation, which relies on backtracking and repeated rollouts, become impractical [^1].
What happened
Changdae Oh, Wendi Li, Seongheon Park, Samuel Yeh, Tanwi Mallick and Sharon Li, from the University of Wisconsin–Madison and Argonne National Laboratory, posted the paper on arXiv on June 24 [^1]. Their claim is that reinforcement learning post-training already leaves behind what a step-level score needs. RL fine-tuning starts from a reference model, such as a base or supervised fine-tuned checkpoint [^1]. For each token an agent produces, the authors compare how likely the trained model finds it with how likely the reference model does. They prove that this log-probability ratio exactly recovers the optimal advantage function, which measures how much better an action is than the average action in that state, even when tool outputs and user replies make the environment unpredictable [^1]. Earlier work had used similar signals in deterministic reasoning settings, an interpretation the authors say doesn't directly carry over to agents [^1]. The authors call the result progress advantage and show it applies both to algorithms with an explicit KL penalty, such as GRPO, and to clipping-based ones, such as DAPO [^1].
The team tested it on Gemma4-4B, Qwen3.5-9B, Qwen3-14B and Olmo3-7B, each paired with its own earlier checkpoint [^1]. In best-of-8 sampling across BFCLv4-MT, WebShop, AgentDojo and τ2-Airline, picking the run with the highest progress advantage averaged 38.8 percent success for Gemma4-4B and 62.1 percent for Qwen3.5-9B. The best other scoring methods reached 33.6 and 55.8, and greedy decoding 33.4 and 54.6 [^1]. Gains were largest where exploratory attempts paid off, such as WebShop, where Qwen3.5-9B went from 21.0 percent with greedy decoding to 42.0 [^1]. It didn't win everywhere: on AgentDojo with Qwen3.5-9B, greedy decoding's 94.8 beat its 91.8 [^1].
The score also predicted whether a τ2-bench customer-service run would succeed. On the Airline domain, its AUROC ranged from 0.720 to 0.865 across the four models, ahead of the trained reward models and confidence baselines [^1]. On the Retail domain, a Claude Sonnet 4.6 judge did better for three of the four models, for example 0.852 against 0.690 with Gemma4-4B [^1]. On Who&When, 184 failed runs from multi-agent systems, the authors say the method rivaled AgenTracer, a model trained to find the decisive error step, on the hand-crafted split [^1].
Why it matters
If the results hold up, the appeal is practical. Anyone with an RL-trained model and its reference checkpoint can score agent runs without collecting step labels or training a separate reward model [^1]. Gemma4-4B's progress advantage also predicted the success of Qwen3.5-9B and Qwen3-14B runs better than confidence baselines did, which the authors read as a sign it could monitor other models' trajectories off the shelf [^1].
The limits are in the paper too. Most model providers don't release intermediate checkpoints, so the method only works where a matching pair is public [^1]. The reference should be neither too close to nor too far from the trained model, and the best way to combine token scores varied: one combination worked best for picking among attempts, another for predicting success [^1]. The paper's conclusion says progress advantage matches or beats a proprietary LLM judge, but its own table shows that on Airline, not on Retail, where Sonnet 4.6 led for three of four models [^1].
Practical example
Say Dana's startup runs a customer-service agent for an airline booking site on Qwen3.5-9B, a model whose base checkpoint, Qwen3.5-9B-Base, is public [^1]. Every night, the team wants to know which of the day's conversations went wrong, without paying people to read them all.
Dana's team feeds each finished conversation through both checkpoints, as the paper did with stored runs, using a single forward pass per conversation [^1]. Wherever the trained model is much more confident than the base model, a step scores high. The paper shows why that matters: in one airline case, plain token probability scored correct tool calls low, while progress advantage scored them positive and rewarded policy terms such as "change of plan" and "business class" [^1].
The lowest-scoring conversations go to a human reviewer first. Within a flagged conversation, the step with the lowest score is the first place to look, the same rule the authors used to locate errors on Who&When [^1]. No labels were collected and no reward model was trained.
There are limits for Dana. If the startup moves to a model whose earlier checkpoint isn't public, the method doesn't apply [^1]. And for a retail-style shop agent, the paper's numbers suggest a strong LLM judge may still flag failures more reliably [^1].
Related gear
We recommend this book because it explains value functions and the policy gradient theorem, the reinforcement learning basics behind the advantage function this paper builds on.