Claw-Eval-Live: No Model Passes 70 Percent of Workflow Tasks
A new agent benchmark builds its tasks from popular ClawHub skills and grades what agents actually did. Of 13 models, the best passed 66.7% of 105 tasks.
TL;DR
- Claw-Eval-Live tests 13 models on 105 workflow tasks built from popular ClawHub skills, grading audit logs and workspace state, not just final answers [^1].
- Claude Opus 4.6 led at 66.7 percent; on service-backed business tasks no model topped 59.8 percent, while terminal repair proved much easier [^1].
Background
AI agents increasingly do multi-step work through harnesses such as Codex, Claude Code and OpenClaw, which let a model call tools, navigate workspaces and query services [^1]. The authors see two weaknesses in many agent benchmarks: they freeze a hand-picked task set at release, so the mix can drift away from what users currently want automated, and they grade mainly the final response, which does not show whether the work was actually done [^1].
What happened
Claw-Eval-Live, from researchers at the Chinese University of Hong Kong, Peking University, Xiamen University and other universities, tackles the first problem by deriving each release from public demand signals [^1]. The current release starts from the 500 most popular skills on ClawHub, ranked by downloads and popularity, which the team groups into workflow patterns and weights into a target mix of task families [^1]. The authors stress that these signals do not measure how often a workflow is used, what it is worth or how hard it is [^1]. From 157 runnable candidates, an optimization step picked 105 tasks that cover all 22 families and separate models well; tasks that every pilot model passed, or every one failed, were excluded [^1]. A release stays fixed once published, and quarterly refreshes are meant to rerun the pipeline [^1].
Of the 105 tasks, 87 run against controlled business services such as CRM, finance, email, calendar, helpdesk and a knowledge base, and 18 are terminal or workspace repairs [^1]. Each service records every read and change the agent makes [^1]. Grading starts from that evidence: whether the agent queried the right sources, whether its entities and numbers match the ground truth, and whether required changes actually happened [^1]. A rubric-bound LLM judge, GPT-5.4, scores only aspects such as completeness or report coherence; the authors note that GPT-5.4 is also one of the tested models, which can bias the judge [^1]. Workspace tasks are graded entirely by scripts [^1]. Agents get a default budget of 24 turns and 300 seconds, a task passes at a score of 0.80, and models are ranked by pass rate, then by average completion [^1].
Claude Opus 4.6 passed 70 of the 105 tasks (66.7 percent), followed by GPT-5.4 at 63.8 percent and Claude Sonnet 4.6 and GLM-5 at 61.9 percent each, down to Doubao Seed 2.0 at 43.8 percent [^1]. The split by surface is stark: every model passed at least 72.2 percent of workspace tasks, but none passed more than 59.8 percent of the service-backed ones, where Opus 4.6 led ahead of GPT-5.4 at 56.3 percent [^1]. HR tasks averaged a 6.8 percent pass rate across models, and no model passed any task in the management family [^1]. Across the leaderboard, 19 tasks were passed by every model and 27 by none [^1].
Why it matters
The results show where the gap is. Agents are already often competent at diagnosing and repairing a local workspace, the authors write; the bottleneck is finishing a workflow across several services "without losing state, missing evidence, or failing to execute required writes" [^1]. The grading also shows what final-answer benchmarks miss: HR onboarding tasks often drew polished but generic write-ups, which failed because they left out employee-specific details or skipped required tool calls [^1].
A single leaderboard number hides a lot as well. MiMo V2 Pro, Kimi K2.5 and Gemini 3.1 Pro all passed 53.3 percent of tasks but differ in average completion [^1]. Costs differ too: estimated API cost for the full run was $31.61 for Opus 4.6 against $6.27 for GPT-5.4, which the authors call the most efficient of the top models, while cheaper models such as DeepSeek V3.2 still trailed the leaders on pass rate [^1]. The authors conclude that deployment choices should weigh accuracy per task family and cost, not a single ranking [^1].
Practical example
Say Tomasz runs operations at a mid-sized logistics firm and wants an agent for month-end work: pull transactions from the finance system, reconcile them against orders and email a summary to the controller. The vendor demo ends with a tidy report, and the model behind it sits near the top of a general leaderboard.
Claw-Eval-Live suggests two checks before Tomasz trusts it. First, the report is not the work. In the benchmark, a fluent summary still scored poorly if the agent skipped required tool calls or missed crucial evidence, because the grader checks the service's audit log, not the prose [^1]. Tomasz's version of that check is to compare the agent's tool log with the finance records: did it read every account, and did the email actually go out?
Second, the right model depends on the task. On the benchmark's e-commerce monthly reconciliation, Claude Opus 4.6 scored 0.94 and Claude Sonnet 4.6 scored 0.07; on a meeting-prep task the order flipped, with 0.05 for Opus 4.6 and 0.91 for Sonnet 4.6 [^1]. So Tomasz runs a few real month-end cases through each candidate model instead of picking from the overall table. If the plan later grows to HR onboarding, the benchmark is a warning: no model passed more than 22.2 percent of its HR and people tasks [^1].
Related gear
We recommend this book because it covers how to build and evaluate applications on foundation models, including agents that call tools, the kind of system Claw-Eval-Live grades.