OpenThoughts-Agent: An Open Recipe for Agent Training Data
A large academic team tested how to build fine-tuning data for AI agents in more than 100 experiments, found the best model is not the best teacher, and released everything.
TL;DR
- The OpenThoughts-Agent team ran more than 100 controlled experiments on building training data for AI agents and released the data, pipeline and models [^1].
- Qwen3-32B fine-tuned on their 100K traces averages 44.8% across seven agentic benchmarks, 3.9 points above the strongest existing open-data model [^1].
Background
AI agents such as Claude Code and Codex work through tasks on a computer, running commands and reading the results [^1]. Open models often learn this from traces: recordings of a teacher model working through tasks step by step, used as examples for supervised fine-tuning. Which tasks and traces make good training data is mostly unpublished. The DeepSeek-V4 paper runs to more than 50 pages but describes its training data in two paragraphs, the authors note [^1].
What happened
The paper, posted to arXiv on June 23 by a large team from UC Berkeley, Stanford, LAION and other institutions, starts from a gap: open agent datasets such as SWE-Smith, SERA and Nemotron-Terminal usually target one benchmark, such as SWE-Bench or Terminal-Bench [^1]. OpenThoughts-Agent aims at data that lifts a model across many agentic tasks. The team split its pipeline into six stages, from sourcing task descriptions to filtering the finished traces, and tested each stage separately: generate 10,000 traces per strategy, fine-tune Qwen3-8B on them and score the result on three benchmarks [^1]. Each such fine-tune took 160 GPU-hours on GH200s. Benchmarks such as Aider Polyglot, GAIA and MedAgentBench were held back until the pipeline was settled, to measure generalization [^1].
The choice of tasks mattered most. Across 95 task-generation strategies, the gap reached up to 30 percentage points on a 100-task SWE-Bench Verified subset and 10 points on Terminal-Bench 2.0 [^1]. Mixing the top 4 to top 8 sources beat the single best one, because coding sets like SWE-Smith mainly lift SWE-Bench, while human-written questions from StackExchange SuperUser lift Terminal-Bench [^1]. The strongest model was not the best teacher: traces from GPT-5.3-Codex led to roughly 5% lower Terminal-Bench 2.0 performance than traces from GLM-4.7-AWQ [^1]. Among the trace filters, dropping traces with fewer than 5 turns gave the largest gain, and the gain held at a matched token budget [^1]. Using an LLM to rewrite, combine or harden task descriptions did not beat leaving them untouched at this scale [^1].
Scaling up exposed a different bottleneck. Generating more traces for the same tasks plateaued between 31.6K and 100K examples, which the team reads as a lack of distinct task descriptions; one top source, human-written cryptocurrency questions from StackExchange Tezos, has only 997 [^1]. Adding sources beyond the top 4 did not reliably help, and 16 sources hurt on every benchmark. What worked was rewriting those 997 Tezos tasks into over 21K distinct surface forms without adding new problems [^1]. The final set, OpenThoughts-Agent-v2, has 100K traces. Fine-tuned on it, Qwen3-32B averages 44.8% across seven agentic benchmarks, against 40.9% for Nemotron-Terminal-32B. On SWE-Bench Verified and Terminal-Bench 2.0 it scores 54.0% and 26.2%, against 41.9% and 25.1% [^1].
Why it matters
The paper's main value is that it is open. It releases the training sets, the pipeline, the experimental data and the models, so other groups can rerun or extend each experiment instead of guessing at choices that model papers leave out [^1]. Some findings cut against intuition: the best-scoring model made a worse teacher, and piling on more traces stopped helping once the tasks stopped being varied [^1].
The team also tried reinforcement learning, but only at 8B scale. After fine-tuning on their data, RL on pymethods2test, competitive-programming problems recast as single Python functions with tests, put a Qwen3-8B model 18 points above its base on average and ahead of the strongest existing models of up to 8B [^1]. The authors list the limits themselves: RL was not tested at 32B, every run started from the Qwen3 family, and nothing beyond 100K traces was tried [^1]. They also call agentic models dual-use and recommend running them with sandboxing and human oversight [^1].
Practical example
Say Priya runs a small machine learning team at a database hosting company and wants an in-house agent for shell and repository chores. Priya's first plan is simple: take the highest-scoring model on Terminal-Bench as the teacher, record it working through tickets from the team's own issue tracker, and fine-tune an open model on every trace.
The paper suggests three changes. First, try more than one teacher on a small set before committing. In the study, GPT-5.3-Codex was the best-performing model but produced weaker training data than GLM-4.7-AWQ [^1]. Second, drop the short traces: removing those with fewer than 5 turns was the strongest trace filter the team found [^1]. Third, don't rely on one kind of task. Coding sets like SWE-Smith mostly lifted SWE-Bench, while human-written SuperUser questions lifted Terminal-Bench, and a mix of the top sources did best overall [^1].
When Priya's tickets run out, recording ever more traces for the same tickets is the obvious next step, and the paper's numbers say it stalls: that approach plateaued between 31.6K and 100K examples, while rewriting existing tasks into new variants kept improving results [^1]. One caveat travels with the recipe. Every run started from a Qwen3 model, and the paper does not test how much the base model contributes [^1].
Related gear
We recommend this book because it walks through pretraining a language model and then fine-tuning it on instruction data, the supervised fine-tuning step this dataset is built for.