QuoteBench: Agent Commands Break When Wrapped in a Second Shell
A new benchmark replays LLM-written Bash commands through one extra double-quoted parser. Success drops by 55 to 73 points, and a matched score can hide that damage.
TL;DR
- QuoteBench shows that routing LLM-written Bash commands through one extra double-quoted shell layer cuts their success rate by 55.4 to 73.2 points [^1].
- Models can partly compensate when told about that layer, so a near-unchanged overall score can hide large opposing effects, and model rankings can flip [^1].
Background
Coding agents run shell commands that a language model writes. Before reaching Bash, that text may be serialized, wrapped and parsed again, for example when a command goes to a remote machine via ssh host "…" or into a container via docker exec sh -c "…" [^1]. Every extra parser re-reads quotes, dollar signs, backticks and newlines, so a command that works on one path can break on another.
What happened
Researchers from Stony Brook University, LMU Munich and the Munich Center for Machine Learning published QuoteBench on arXiv on August 13 [^1]. It has 56 one-shot Bash tasks in 14 operation families, each with one harmless control and three variants with hazardous payloads: multiline text, hostile filenames, regex and glob characters, heredocs, literal arguments and environment values, Git metadata and two simulated SSH-like hops [^1]. Validators check the final state, such as exact file bytes, argument vectors, JSON or Git history, so any command that reaches the right state passes. Exit codes would not do: 23.4 to 47.0 percent of failing runs exited with zero while leaving the wrong state [^1]. The families come from a survey of 86 incidents in the authors' own agent sessions and 412 screened public reports [^1].
The core experiment keeps the model's reply fixed and changes only how it reaches Bash. On the raw path, the reply runs directly; on the "nested" path, it is first interpolated into bash -c "…", adding one double-quoted parser of the kind found on remote, container and CI paths [^1]. A second prompt, differing by one sentence, tells the model about that boundary without giving quoting advice [^1]. Across eight configurations queried on July 31, the extra parser cut success by 55.4 to 73.2 points. Even the 14 harmless control tasks lost 28.6 to 57.1 points, because models use characters that are active inside double quotes in ordinary commands [^1]. Telling the model about the boundary recovered 30.4 to 60.7 points for six configurations; for Qwen3.5-27B and Gemini-3.1-Flash-Lite the effect was zero or slightly negative [^1].
The fixes are simple. Escaping the reply at the point where it is interpolated, or running it as a temporary script, reproduced the raw-path outcome for all 448 public task pairs [^1]. ShellCheck, a static checker for shell scripts, flagged only 34.6 percent of the failures caused by nesting, because each command was well-formed on its own and the fault lay in the later interpolation [^1].
Why it matters
The paper's main target is how agents are scored. Raw command generation is close to saturated at the frontier: six frontier configurations passed 91.1 to 100 percent of tasks on the direct path [^1]. GPT-5.6-sol's score moved only 3.6 points between the plain setup and the disclosed nested one, but that small gap is the sum of a 64.3-point loss from the transport and a 60.7-point recovery from the model adapting [^1]. Rankings move with the deployment path too. GPT-5.6-sol trailed Gemini-3.5-Flash by one task on the raw path and led it by eighteen on the nested one, the one unambiguous reversal among 26 comparable pairs, with four more on single-task margins [^1].
The authors therefore ask that evaluations of command-issuing agents report the model configuration, generation contract, execution path, operating point and final-state validator, rather than treat a matched score as a property of the model [^1]. They stress that the contribution is the measurement, not the fix: both fixes require the caller to control the boundary, and their survey of six public agent systems found boundaries applied downstream of the stated contract [^1]. The benchmark covers POSIX and Bash only, and because hosted models can change under the same name, the query date is part of the result [^1].
Practical example
Say your coding agent writes a command that puts a line containing a dollar sign into a config file, and it works when you test it locally. In production, your deploy script sends the same command to a server inside ssh host "…", so the text is parsed once more inside double quotes [^1]. Now the dollar sign gets expanded, the file holds the wrong text, and the command may still exit with zero, as many failing runs in QuoteBench did [^1]. The paper's two fixes apply directly: escape the command where you wrap it, or ship it as a temporary script [^1]. Then check the file itself, not just the exit code.
Related gear
We recommend this book because it explains how the shell expands and quotes what you type, the rules behind the failures QuoteBench measures.