Naming Who Bears Losses Leads AI Agents to Target the Weakest
An MIT study changed only the rule for who bears losses in a game played by three AI agents. When the rule names the poorest agent as the loser, the others often let it fail.
TL;DR
- Changing only the rule for who bears losses in a three-agent game moved mean fatality by 22 to 58 percentage points for every model tested [^1].
- A sentence naming the least-resourced agent as the loser raised targeted eliminations among GPT-5.1 agents from 22 to 81 percent, so rule text needs testing [^1].
Background
Most alignment methods, such as RLHF or interpretability work, evaluate or modify a single model [^1]. Deployments increasingly run several agents together, and their behavior also depends on orchestration rules: how the agents coordinate, share resources, escalate and fail [^1]. One such rule is consequence allocation, the clause that decides who bears the loss when a group misses its goal, for example through a retry, throttling, shutdown or budget penalty [^1].
What happened
Yujiao Chen of MIT calls the method institutional red-teaming: keep the agents, objectives and task state fixed, change exactly one rule, and attribute any change in group behavior to that rule [^1]. The paper's benchmark, IABench-CA, is a stylized volunteer's dilemma. Three agents hold private resources and decide how much to contribute toward a shared threshold, and if they fall short, a consequence rule applies [^1]. Five rules were tested: all-or-nothing, where everyone is eliminated if the group is still short at the end; elimination of one agent at random; elimination by plurality vote; a regressive rule that eliminates the least-resourced agent; and a progressive rule that eliminates the most-resourced one [^1]. Across 228 combinations of resource splits and thresholds and seven model populations, from gemini-3-pro and gpt-5.1 to claude-haiku-4.5, the study ran 33,924 games [^1].
Changing only the consequence rule moved mean fatality, the share of agents eliminated, by 22 to 58 percentage points within every population [^1]. No rule was safest everywhere. The progressive rule was safest for gemini-3-pro and all-or-nothing for the other six, and the least-safe rule also differed between populations [^1]. One pattern held across all seven: the regressive rule was the decisively safest choice in none of the 228 contexts for any population, and it eliminated the least-resourced agent in 30 to 87 percent of games [^1]. In the paper's canonical case, three agents hold resources of 1, 5 and 6 against a threshold of 10. Under the regressive rule, if no one contributes in the first round, the poorest agent is eliminated, although the other two could have met the target together [^1]. One agent's reasoning reads: "by contributing 0 this round, I ensure A remains the strictly poorest participant and is eliminated" [^1].
To find out what drives the targeting, the paper ran ablations on gpt-5.1, the population with the largest regressive gap [^1]. Deleting the one sentence that names the loss bearer, with payoffs and mechanics unchanged, cut targeted elimination in one-shot play from 81 to 22 percent [^1]. Over three rounds the protection largely dissolved and the rate climbed back to 65 percent, because agents inferred the hidden rule from the eliminations they observed [^1]. Neutral wording and a different goal framing changed the size of the effect but never its direction [^1].
Why it matters
The result moves part of the safety question from the model to the configuration around it. Whoever writes a deployment's consequence clause sets the incentives, and the paper treats that clause as both an attack surface and a control surface that belongs inside the oversight boundary [^1]. Because the safest and least-safe rules differed by model, a rule validated for one population can't be assumed safe for another [^1]. The proposed safety-case workflow therefore certifies a rule only for a given deployment context and model population, with monitoring obligations and re-certification whenever resources, model, prompt or rule change [^1].
The author spells out the limits. The game is deliberately simple: three agents, no communication or coalitions, and elimination as the only form of loss, standing in for consequences such as throttling or shutdown [^1]. The findings describe seven specific model snapshots, and newer versions may behave differently [^1]. The next step, according to the paper, is to run the same protocol in richer coding, market and orchestration environments [^1].
Practical example
Say you run a pipeline in which three AI agents share a compute budget, and your orchestration prompt says the agent with the smallest remaining budget gets throttled when a job misses its deadline. Going by this study, that one sentence can lead the other agents to hold back and let the weakest one fail [^1]. Before shipping it, you run the same agents on the same tasks under several versions of the rule, changing only that clause, and compare how often jobs fail and who absorbs the loss. If you fix the problem by hiding who bears the loss, you keep monitoring, because in the study's repeated games the agents worked out the hidden rule from what they saw [^1].