MAS-PromptBench: Prompt Tuning Gains Shrink in Agent Teams
A Johns Hopkins benchmark tests automatic prompt optimization on multi-agent LLM systems. Gains reached 24 points in one setup, but averaged less than for a single model, and some setups got worse.
TL;DR
- MAS-PromptBench measures how much automatic rewriting of system prompts helps teams of LLM agents across nine tasks, five workflows and four frameworks [^1].
- Results swung from a 24.0-point gain to a 16.0-point loss, and the average gain fell as teams grew [^1].
Background
A multi-agent system splits a task among several LLM-based agents. Each one gets a system prompt that defines its role, plus a place in a workflow that decides who talks to whom and how answers are combined [^1]. Automatic prompt optimizers, which rewrite prompts based on feedback from test runs, have shown strong gains for single models. The authors say it was unclear whether that carries over to teams, where the search space grows exponentially with the number of agents [^1].
What happened
Juyang Bai and Laixi Shi of Johns Hopkins University posted MAS-PromptBench on arXiv on June 22 [^1]. It builds agent teams in LangGraph, CrewAI, AutoGen and the OpenAI Agents SDK and runs them on nine tasks: reasoning (GPQA-Diamond, HotpotQA, MATH), coding (LiveCodeBench, APPS, SWE-Bench Verified) and tool calling (BFCL, ToolHop, API-Bank) [^1]. The benchmark varies the workflow (single agent, independent, sequential, centralized with a coordinator, or decentralized), the format of messages between agents (free-form, semi-structured or JSON-style structured), team size (2, 4, 8 or 10 agents) and the optimizer [^1]. The two optimizers are multi-agent versions of GEPA and MIPRO, which rewrite one agent's prompt at a time using feedback from the team's runs [^1]. Every agent ran on Qwen3.5-9B with thinking mode disabled, while Qwen3.5-122B-A10B-FP8 proposed the prompt edits. A new prompt set was kept only if it beat the original on a 25-example validation split [^1].
The results were mixed. Averaged over workflows, the GEPA-based optimizer improved seven of nine tasks, with the largest average gain of 10.0 points on APPS [^1]. Single setups went further: a sequential chain on BFCL gained 24.0 points. Independent agents on MATH lost 16.0 points, falling from 76.0 to 60.0 percent [^1]. Coding tasks gained 3.7 points on average and tool calling 4.3, against 1.3 for reasoning [^1]. In the workflow study, no multi-agent workflow averaged more than 2.3 points, while the single-agent baseline gained 4.2. Independent agents lost 0.5 points on average [^1].
Two other factors stood out. Average gains rose from 1.6 points with free-form messages to 2.4 with semi-structured and 4.3 with structured ones, and the MIPRO-based optimizer showed the same order at 0.1, 4.8 and 6.3 points [^1]. Team size worked the other way: the average gain was 2.4 points with two agents, 0.6 with four, minus 0.9 with eight and minus 2.1 with ten [^1].
Why it matters
The study suggests that gains from prompt optimization on one model can't be assumed to carry over to a team built from it. The authors' hypothesis is that code and tool calls produce verifiable artifacts and clear interfaces, so a better prompt for one agent survives the handoffs. In reasoning chains, local improvements are "discarded, overwritten, or amplified before reaching the final answer" [^1]. For parallel agents, they suggest that "uncoordinated prompt revisions" may "erase one another's gains" [^1]. The safeguard in the setup did not prevent regressions either: prompts were adopted only after beating the originals on validation examples, and test scores still dropped in several configurations [^1].
The scope is narrow. All agents used one 9-billion-parameter model without thinking mode, so the authors say the scores reflect "a controlled agentic protocol rather than the maximum achievable performance" of the model [^1]. They also name only two optimizers as a limitation, and call for optimizers that take task structure and team configuration into account [^1].
Practical example
Say Lena runs support tooling at a small software company. Lena's team built a centralized setup: a coordinator agent splits each customer question among worker agents that search the documentation, then merges their findings into one answer. Like HotpotQA, the questions often need facts from several pages [^1].
To improve it, Lena runs a GEPA-style optimizer that rewrites each agent's prompt in turn and keeps the result because it scored better on a small validation set [^1]. The benchmark shows how this can backfire. On HotpotQA with a centralized team, optimization raised exact-match accuracy by 5.0 points with two agents but cut it by 9.0 points with four and by 12.0 with ten [^1]. A coordinator, the authors note, tends to amplify both successes and failures [^1].
The paper points to two changes before Lena trusts the new prompts. First, give the agents a fixed message format with fields such as status, summary, confidence, evidence and next action. In the paper's protocol tests, a centralized HotpotQA team gained 10 points under structured messages, against 1 point with free-form text [^1]. Second, keep the team small and test the optimized prompts on fresh questions, since a win on 25 validation examples did not guarantee a win at test time [^1].
Related gear
We recommend this book because it covers prompt engineering and agents for applications built on foundation models, the two topics this study brings together.