Training Agents Against One Simulated User Fails With Real People
Researchers show that RL against a single frozen LLM playing the user narrows the agent onto the simulator's habits. Two fixes lifted held-out success by up to 14 percent.
TL;DR
- Training conversational agents with reinforcement learning against one frozen LLM playing the user makes them overfit to that simulator's habits, a new paper shows [^1].
- Two fixes, sampling varied simulator replies or training the simulator alongside the agent, improved results on held-out simulators and with real participants [^1].
Background
Reinforcement learning trains a model, called the policy, by rewarding conversations that end well. For multi-turn tasks such as customer support or persuasion, training with real users at scale is too expensive and slow, so a common recipe prompts a single frozen language model to play the user [^1]. Aligned models tend to give typical answers, and earlier studies found LLM-simulated users overly cooperative and stylistically uniform [^1].
What happened
Researchers from Northeastern University, New York University, UC Berkeley, the University of Washington and Stanford posted the paper to arXiv on August 12 [^1]. They call the failure "simulator collapse." Because the simulator keeps producing its most likely response, the policy's training signal is dominated by that mode, and the policy overfits to narrow strategies that exploit it [^1]. During training, the policy's entropy, a measure of how varied its behavior is, drops to near zero. The result performs well against its training simulator but transfers poorly to unseen simulators and real users [^1].
The team tested this on three benchmarks: Persuasion for Good, where an agent argues for a charitable donation against a resistant donor; τ²-bench, with customer-service dialogues for retail and airline tasks; and CooperBench, where two coding agents coordinate on a task [^1]. The standard recipe used GPT-5-mini as the single simulator. With Qwen3-4B-Instruct as the policy, its best checkpoint reached 46.1 percent success on the retail split against 40.4 for the untrained model, but that was a transient peak that collapsed back toward the baseline before training ended [^1].
The paper proposes two fixes. Verbalized Sampling leaves the simulator frozen but asks it at each turn for several plausible replies with their likelihoods, then samples one, so the policy meets acceptance, pushback and refusal [^1]. Co-Training updates a trainable simulator alongside the policy, so the behavior the policy would lock onto keeps moving; a population variant draws the simulator from a pool of recent checkpoints [^1]. On Qwen3-4B-Instruct, Verbalized Sampling raised retail success to 55.5 percent and Population Co-Training to 62.2 percent [^1]. In a human study with 40 Prolific participants per condition, the τ²-bench task outcome rose from 0.43 under single-simulator training to 0.63 with Verbalized Sampling and 0.70 with Co-Training, and both fixes made the persuasion dialogues rate as more natural [^1].
Why it matters
The authors locate the problem in the training environment, not the learning algorithm, and say that is why both fixes are simple [^1]. The paper also exposes a measurement trap: reported as a best checkpoint, single-simulator training looks better than the untrained model on every benchmark cell, while its training curve on the retail split falls back toward that baseline [^1]. A team that only reads the headline number would miss the collapse. The fixes have a price. Co-Training costs roughly twice as much compute per step, while Verbalized Sampling recovers most of the gain at matched compute [^1].
The authors are clear about the limits. Their held-out panel consists of aligned LLMs that share biases with the training simulators, which is why the human study is the direct test of real-user transfer, and all settings are text-only, two-agent and in English [^1]. They also conjecture that similar collapse could appear in reasoning, code and tool-use training whenever the grader is itself a mode-collapsed LLM [^1]. Their framework for this kind of training, SCOPE, is released as open source [^1].
Practical example
Say you are fine-tuning a support chatbot with reinforcement learning and use one LLM to play the customer. Going by this paper, the warning sign is not a bad score but a curve that peaks early and sinks back toward where you started, while the bot's replies become ever more uniform [^1]. Before shipping, you test against simulators from other model families the bot never trained with [^1]. The cheapest fix the authors describe needs no retraining of the simulator: at each customer turn, ask it for several plausible replies with likelihoods and sample one, so the bot also meets customers who push back or refuse [^1].
Related gear
We recommend this book because it explains policy-gradient methods such as REINFORCE, the update rule the paper's training builds on.