inferwire
/
AI·2 min read

NVIDIA Method Predicts Which Base Models Make Good Coding Agents

Researchers from NVIDIA, the University of Minnesota and UC Berkeley score untuned base models at the decisive step of solved coding tasks, and the scores rank models like post-trained SWE-bench results.

TL;DR

  • Deciding which base model is worth an expensive round of agentic post-training is hard, because untuned models usually cannot run a coding agent's tools [^1].
  • An NVIDIA-led team scores base models at the "decisive step" of known solutions; across ten model pairs, the rankings closely matched post-trained SWE-bench Verified results [^1].

Background

Coding agents are built by post-training a base model to work in a tool harness, editing a repository step by step until the tests pass. That post-training is costly, so labs want to know in advance which checkpoint will pay off [^1]. End-to-end benchmarks such as SWE-bench Verified are too demanding for base models, whose pass rates stay near zero; single-shot coding tests avoid that, but they skip the core skill of keeping track of a repository over many steps [^1].

What happened

Researchers from NVIDIA, the University of Minnesota and the University of California, Berkeley use successful trajectories of already post-trained agents as a "lookahead signal" [^1]. They replay each trajectory, rerun the tests after every code-changing step and find the decisive step: the first one whose cumulative patch flips the repository from failing to passing [^1]. At that point they apply three screens that do not need the base model to drive the harness itself [^1]. One measures how much probability the base model assigns to the action that solved the task. Another asks it to choose between that action and alternatives the same tests rejected. The third lets it continue from the prefix and counts any continuation the tests accept [^1].

Across ten pairs of public base and post-trained models, all three screens ranked the models in close agreement with post-trained SWE-bench Verified pass@1 [^1]. Because the method needs only a benchmark's successful trajectories and its verifier, the authors say it can turn future agentic coding benchmarks into evaluations for base models [^1].

Why it matters

Choosing a base checkpoint is a high-stakes decision made before its agentic abilities can be seen, and revealing them takes substantial compute and engineering [^1]. A cheap screen that predicts the outcome lets a lab drop weak candidates early and spend post-training budget where it will show up in results.

Practical example

Say a lab has four candidate base models and budget to post-train one into a coding agent. Instead of training all four, it replays solved SWE-bench tasks, stops each at the decisive step and checks which model most strongly prefers the correct fix. According to the paper, that ranking closely tracks how the models would do after post-training [^1].

Related gear

We recommend this book because it offers a comprehensive guide to designing and evaluating machine learning systems, which aligns perfectly with the structured validation methodologies analyzed in this post.

AdvertisementAmazon

Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications

Sources

  1. [1]arXiv — Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models