ExpRL Turns Reference Solutions Into Rewards for LLM Training
ExpRL hides human-written solutions from a language model and has an LLM judge use them to grade partial progress, giving standard RL a better starting point on hard math.
TL;DR
- ExpRL hides human-written reference solutions from a language model and has an LLM judge use them to grade the model's partial progress during RL [^1].
- Used to prime a 4B model before standard RL, ExpRL beat fine-tuning, sparse-reward RL and self-distillation on all four held-out math benchmarks [^1].
Background
Reinforcement learning with sparse rewards, where a large language model (LLM) earns credit only for a correct final answer, has become a standard way to improve reasoning [^1]. It only works if the model already samples useful solution paths: on problems it never solves, there is nothing to reinforce [^1]. Models are therefore often mid-trained first, usually by fine-tuning on curated reasoning traces, which means deciding by hand what the model should learn [^1].
What happened
In a paper posted to arXiv, Violet Xiang, Amrith Setlur, Chase Blagden, Nick Haber and Aviral Kumar propose ExpRL, short for Exploratory RL: RL-based mid-training on large collections of human-written questions and answers [^1]. Instead of training the model to imitate a reference solution, ExpRL hides it. The model being trained, the policy, sees only the original problem, and an LLM judge compares each attempt with the reference under a fixed rubric, scoring it from 1 to 5 [^1]. The judge is told to verify rather than solve, and not to fill in missing steps [^1]. ExpRL-Outcome scores the complete attempt. ExpRL-Process also scores each partial attempt, cut at a delimiter the model emits between reasoning steps, and rewards a segment only if it raises the score over the previous one [^1].
The main experiments use Qwen3-4B-Instruct-2507 as both policy and judge, trained on hard math problems the base model failed to solve in 64 attempts each [^1]. After 230 steps of ExpRL, each model gets 500 steps of ordinary sparse-reward RL, so only the starting point differs [^1]. Measured by pass@1, the share of single attempts that are correct, ExpRL variants came out on top on all four held-out benchmarks. On AIME 2026, ExpRL-Process reached 63.41 percent, against 58.75 for the sparse-reward GRPO baseline and 51.40 for the untouched base model [^1]. Supervised fine-tuning (SFT) on the same reference solutions ended at 30.26 percent, below the base model [^1].
During ExpRL training, the model's token-level entropy, a measure of how varied its outputs are, stayed stable or rose slightly, while sparse-reward GRPO's dropped the most and unlocked the fewest unsolved prompts [^1]. Annotated attempts showed more verification, self-correction and backtracking than the base model; SFT lost verification behavior [^1]. A second experiment trained a larger Qwen3-8B with a Qwen3-4B judge on 4,001 math, science and coding examples. ExpRL-Outcome improved every pass@1 score, but on the coding benchmark LiveCodeBench, sparse-reward GRPO stayed ahead [^1].
Why it matters
The result changes how human-written solutions are best used. The authors conclude that, for priming a model before RL, reference solutions "are more effective when used to score sampled reasoning than when used only as trajectories to imitate" [^1]. The setup can also be cheaper than expected: because checking an attempt against a reference is easier than writing a solution, a 4B judge could provide useful rewards for an 8B model [^1].
The limits are clear, and the authors name them. ExpRL needs reference solutions, which may be hard to get in some domains [^1]. The judge must be capable and see the right reference. In a calibration test, a Qwen3-4B judge's misplacement rate, the average of its false-positive and false-negative rates, was 17.8 percent on math with the correct reference, 39.2 without one and 50.4 with a reference from another problem; a 0.6B judge was unstable [^1]. For coding, where running the code gives a strong reward, the authors say downstream RL should use that signal directly [^1]. All trained models are Qwen3 models with 4 or 8 billion parameters [^1]; the paper does not test larger ones.
Practical example
Say Priya runs a small lab that wants a 4B open model to solve olympiad-style math. The lab owns a few thousand worked solutions written by human tutors, and the model fails the hardest problems every time. Plain sparse-reward RL gives each of those attempts a reward of zero, so the problems teach the model nothing [^1].
Priya's first idea is to fine-tune on the tutors' solutions. The paper warns against exactly that: in its tests, SFT on reference solutions dropped the 4B model's AIME 2025 pass@1 from 46.46 to 6.00 percent before any RL [^1]. Human-written solutions differ substantially from the model's own reasoning, and cloning them can disrupt it [^1].
With ExpRL, Priya keeps the solutions away from the model. For each problem, the model writes its own attempts, and a judge model reads each one next to the tutor's solution. An attempt that picks the right approach but fails later still earns partial credit; one that wanders off earns little [^1]. After this priming stage, Priya runs ordinary RL with right-or-wrong rewards, starting from a model that now gets more hard problems right [^1].
Two checks matter. Each problem must be paired with its own solution, since a reference from another problem makes the judge's scores unreliable [^1]. And for the lab's coding tasks, rewards from actually running the code remain the better signal [^1].
Related gear
We recommend this foundational text because it provides the core mathematical principles of reinforcement learning that the researchers are now applying to mid-training for language models.