Final Leaderboard Rankings Hide the History That Matters
A paper rebuilds the public histories of LiveBench, Open LLM Leaderboard v2 and LMArena, shows a final table can fit very different stories, and rejects its own model in the audit.
TL;DR
- Yanan Long rebuilt the public histories of LiveBench and Open LLM Leaderboard v2 from 94 and 262 snapshots, arguing that final rankings discard evidence [^1].
- The paper's own selection-aware statistical model failed all four of its pre-registered tests, which the author says shows the audit can reject unsupported claims [^1].
Background
AI leaderboards such as LiveBench, the Open LLM Leaderboard and LMArena publish rankings that change as systems are added and benchmarks revised [^1]. Most readers see only the latest table. The paper argues that each table is selective: it shows a ranked subset and hides systems that were tried, unpublished, revised or below a threshold [^1]. A final ranking is therefore "a lossy endpoint of a repeated public history, not the history itself" [^1].
What happened
The paper, posted to arXiv on June 15 by Yanan Long of StickFlux Labs, builds an archive from repeated public snapshots [^1]. Each record stores the source, timestamp, scores, score direction, how ranks and duplicates are handled, and what is missing. Sources that fail these checks are excluded rather than silently entering the analysis [^1]. LiveBench (94 snapshots, 195 systems) and Open LLM Leaderboard v2 (262 snapshots, 4,484 systems) are the primary archives. LMArena (152 snapshots, 365 systems) serves as a stress test for preference ratings, and GAIA and tau-bench as limited pilots for agent benchmarks [^1]. LiveCodeBench, HELM Capabilities and SWE-bench Verified were left out because their public histories lacked the versioned source tables the method needs [^1].
The central result is a constructed example. With 1,000 systems and a top-10 table, two different histories produce exactly the same final table, yet imply that the score frontier comes within 0.05 of the ceiling at time 23.03 in one case and 75.13 in the other, on a scale where the final snapshot sits at time 10 [^1]. The author stresses that this is one admissible example, not a general result [^1]. In synthetic tests of a decision layer with five possible actions, from "continue" to "collect" more data, the action chosen from the full history and the one chosen from the final snapshot alone agreed at rates of 0.82, 0.76 and 0.94 across three scenarios [^1].
The author then audits a candidate model that accounts for the reporting cutoff instead of treating published leaders as ordinary results, with pass rules fixed before the results were inspected [^1]. It failed every gate. It passed the known-truth recovery test in 0 of 3 synthetic settings, though it correctly avoided a false plateau in a slow-progress control [^1]. On LiveBench and Open LLM Leaderboard v2 it predicted later snapshots worse than both required comparison models, beating only a rolling-maximum heuristic; on LMArena it trailed the standard Bradley-Terry/Elo rating on every metric, by modest margins; and its uncertainty estimates were badly calibrated [^1].
Why it matters
The practical lesson is about claims of saturation. If one final table fits histories with very different timing, a statement that a benchmark is about to plateau needs the earlier snapshots, with benchmark versions and visibility rules recorded [^1]. The paper places this next to earlier work it cites, which found that leaderboard rank can diverge from user utility and that small benchmark differences are often statistically fragile [^1]. For agent benchmarks the gap is larger still: the GAIA and tau-bench pilots exposed weak scaffold and tool metadata, and the paper says such archives also need tool budget, environment version, retry policy, judge version and human-intervention policy [^1].
What the paper does not deliver is a working forecaster. Its positive results concern the archive and the audit procedure. The decision losses are stylized rather than elicited, no prior or loss robustness sweep was run, and the calibration checks rest on only 20 and 11 ranks [^1]. The timing example also depends on conventions the author chose, such as a ceiling of 1 and a threshold of 0.05 [^1].
Practical example
Say Tomasz leads a small team that picks open models for an internal coding assistant, and has to decide which benchmarks to keep in the team's evaluation suite. Tomasz opens the latest Open LLM Leaderboard v2 table, sees the top scores bunched together and concludes the benchmark is nearly used up.
The paper shows why that step is shaky. A final table only constrains the final snapshot: in the author's example, the same table fits a history in which the gap closes at time 23.03 and another in which it takes until 75.13 [^1]. To judge saturation, Tomasz needs the earlier snapshots, the benchmark version each one used, and the rule for which systems were shown, the metadata the paper's archive records [^1].
A fancier statistical model is no shortcut either. The paper's own selection-aware model predicted later snapshots worse than its comparison models on both primary archives [^1]. So before trusting any tool that forecasts benchmark progress, Tomasz tests it the way the paper does: fit it on the snapshots up to some date, forecast the next one, and compare it with a plain baseline [^1]. If the team also tracks agent leaderboards, Tomasz first checks whether each entry records its scaffold and tools, since the paper found that metadata weak in GAIA [^1].
Related gear
We recommend this book because it explains Bayesian reasoning and why so many forecasts fail, the two themes this paper applies to AI leaderboards.