AI4AI-Bench: Coding Agents Rarely Improve How Models Learn
A new benchmark gives coding agents four hours to improve the training algorithm in ten research repositories. The best system closed under a fifth of the gap to the optimum.
TL;DR
- AI4AI-Bench gives coding agents four hours to improve the training algorithm in ten research repositories, then retrains their code from scratch and scores it [^1].
- The best system, Claude Opus 5 under Claude Code, averaged 0.250 on a scale where 0.1 is the original algorithm and 1.0 the optimum [^1].
Background
Recursive self-improvement is the idea that an AI system could improve the process that produces AI systems [^1]. The authors, from Einsia.AI and Tsinghua University, see the training algorithm as the decisive part: a better objective or update rule improves every later run, as Adam, DPO and GRPO did [^1]. Existing benchmarks, they say, are won by collecting data or tuning hyperparameters and don't separate how a run is executed from how the model learns [^1].
What happened
The benchmark freezes ten research repositories covering ten families of training algorithms, from supervised fine-tuning and multi-turn agentic reinforcement learning to machine unlearning, weight averaging and one-shot pruning [^1]. In each task, the agent can read the code, change it and test ideas against a cheap proxy metric for four hours. What it leaves behind is a source-code patch, which is run from scratch for up to 12 hours and scored against the repository's original algorithm under the same procedure [^1]. Because the ten metrics can't be compared directly, every task is mapped onto one scale: 0 is an uninformative model, 0.1 the algorithm the repository ships and 1.0 the task optimum [^1].
The study tested six systems, each a combination of model, harness and reasoning effort: GPT-5.6 Sol, Terra and Luna under Codex, Claude Opus 5 and Sonnet 5 under Claude Code, and Kimi K3, also under Claude Code [^1]. Across their 29 configurations, the mean score was 0.166. Claude Opus 5 led at 0.250, followed by GPT-5.6 Sol at 0.191, Kimi K3 at 0.174, Claude Sonnet 5 at 0.145, GPT-5.6 Terra at 0.135 and GPT-5.6 Luna at 0.117. The single best configuration, Opus 5 at medium effort, averaged 0.288 [^1]. In 124 of the 290 attempts, the agent left the repository worse than it found it [^1].
Why it matters
The paper's analysis shows where the gap comes from. Of the 263 submissions that changed anything, 141 left the learning procedure untouched and adjusted budgets, checkpointing, hyperparameters or model capacity instead [^1]. The 122 that changed the objective, the supervision signal, the learning rule or the data averaged 0.226, against 0.126 for the rest [^1]. More reasoning effort mainly made agents willing to go there: it raised the share of such submissions from 8 to 64 percent and the mean score from 0.094 to 0.196 [^1]. Spending didn't explain the ranking. A median configuration cost $434 for Sol and $48 for Luna, while Opus 5 led at a median of $181 [^1].
For claims that agents will soon speed up AI research by improving their own training, the result sets a baseline: even the strongest system closed under a fifth of the distance between the existing algorithm and the optimum [^1]. The authors release the tasks, the evaluators and every scored submission, so the measurement can be repeated as systems change [^1].
Practical example
Say you hand a coding agent your team's fine-tuning repository and ask it to improve the results. Going by this benchmark, the likely outcome is a patch that changes budgets, hyperparameters or the checkpoint schedule rather than the loss or the update rule [^1]. Before accepting it, you check which kind of change it is, retrain from a clean start and evaluate on data the agent never saw, as AI4AI-Bench does. In more than two fifths of the benchmark's attempts, that step would have shown a result worse than the original [^1].
Related gear
We recommend this book because it explains reinforcement learning from the ground up, one of the families of training algorithms the benchmark asks agents to improve.