inferwire
/
AI·4 min read

Coding-Agent Speed Benchmarks Lose Signal on Other Machines

Replaying 740 reference patches on four cloud machine types, researchers found that only 11 of 140 SWE-Perf tasks stay valid, and that scoring rules reshuffle leaderboard rankings.

TL;DR

  • Reference patches stayed valid across all tested cloud machines in only 39/102 GSO, 11/140 SWE-Perf and 411/498 SWE-fficiency tasks [^1].
  • Rankings also depend on scoring: for eight shared submissions, GSO and SWE-fficiency disagree on 9 of 28 head-to-head orders, so rank alone doesn't measure skill [^1].

Background

Repository-level performance benchmarks such as GSO, SWE-Perf and SWE-fficiency ask a coding agent to make a real project faster without breaking it [^1]. Unlike bug-fixing benchmarks such as SWE-Bench, which mainly check whether tests pass, they have to compare runtime measurements, and those fluctuate with CPU scheduling, cache state and memory bandwidth contention [^1]. Each task comes with a reference patch, an official optimization that submissions are measured against [^1].

What happened

Zhi Chen, Zhensu Sun, Yuling Shi, David Lo and Lingxiao Jiang of Singapore Management University and Shanghai Jiao Tong University audited the three benchmarks [^1]. They replayed all 740 official reference patches on four Google Cloud machine types with 64 vCPUs and 256 GB of memory each, covering older and newer Intel and AMD processors, for three rounds per machine [^1]. Almost every task still ran. But only 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks and 411 of 498 SWE-fficiency tasks met their benchmark's own validity rule in all 12 replays [^1]. SWE-Perf's problem isn't extra noise but a tiny signal: its median reference patch changed runtime by only -0.03%, so small timing shifts can wipe out the statistical support [^1].

Next, the team looked at how task results become rankings, using the eight public submissions that appear on both the GSO and SWE-fficiency leaderboards, all built on the OpenHands agent scaffold [^1]. The two official rankings disagree on 9 of the 28 head-to-head pairs, and the GPT-5-labeled submission moves five places [^1]. GSO counts a task as solved only if a correct patch matches or beats the reference speedup. SWE-fficiency averages speedup ratios with a harmonic mean and a floor of 0.001, which lets a few failures dominate: the worst ten tasks carry 58.5 to 82.8 percent of a submission's score weight [^1]. In the Claude Opus 4.5 submission, one near-floor task alone carries 33.6 percent. Capping the penalty for a single task, which the authors tested as a diagnostic, changed six of eight ranks [^1].

Finally, the authors pooled 10 public submissions per task. Of 450 replay-valid GSO and SWE-fficiency tasks, at least one submission matched or beat the reference on 384, every task had a passing patch, and 449 had one that was faster than the unoptimized code [^1]. For the remaining 66, the best public patch reached a median 85.3 percent of the reference speedup on GSO and 87.9 percent on SWE-fficiency [^1]. The authors read this as a gap in implementation depth: agents often find a useful optimization idea but don't apply it broadly or carefully enough [^1].

Why it matters

Scores on these benchmarks are increasingly used as evidence of coding-agent progress, and the authors argue they should not be read as direct labels of agent capability [^1]. Some tasks measure a difference that is hard to tell apart from runtime noise, and on SWE-fficiency a rank gap may reflect broad performance, a handful of severe failures, or both [^1]. The authors ask benchmark reports to separate tasks verified across machines from unstable ones, show how much each task weighs in a score, and compare submissions under scoring rules that fit the use case [^1].

The pooled results also narrow what the benchmarks can still show. With most tasks already solved by some public submission, a new agent configuration judged only on extra reference-level successes has little room left, and its measured gain may hinge on a few tasks [^1]. For future benchmarks, the authors want the harder part of real performance work: finding the bottleneck from profiles or traces, validating against workloads the agent never saw, and measuring latency and memory, not just one runtime number [^1].

Practical example

Say your team is choosing between two coding agents for performance work, and one ranks three places higher on SWE-fficiency. Before trusting the gap, you check the per-task outputs: if a few near-floor tasks carry most of the score weight, the ranking may reflect those failures more than broad speedups [^1]. You also check whether the tasks behind the difference have reference patches that stay valid across machines [^1]. Then you run both agents on a slow function from your own codebase and time the results several times on your production hardware, because runtimes shift between runs and machines [^1].

Related gear

We recommend this book because it explains how to measure and profile the performance of real systems, the diagnostic work the study says today's benchmarks leave out.

AdvertisementAmazon

Systems Performance

Sources

  1. [1]arXiv — Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?