OmegaUse-OfficeVal: Office Agents Are Cheap but Trail Humans
Baidu researchers built 100 long office tasks, each tagged with human labor time and a price. The best model scored 17.91 against 27.79 for humans, at a fraction of the cost.
TL;DR
- Baidu's OmegaUse-OfficeVal pairs 100 long office tasks with recorded human labor time and an estimated price, so agents can be weighed against workers [^1].
- The best model, GLM-5.2, scored 17.91 against 27.79 for humans, and every model tested was cheaper and faster but produced weaker files [^1].
Background
Office agents are LLM systems that work on documents, spreadsheets, presentations and PDFs for a user. According to the authors, existing office benchmarks mostly test short, bounded operations or synthetic workflows, while broader productivity benchmarks such as GDPVal and the Remote Labor Index tie economic value to occupational categories rather than single tasks [^1]. The question they want answered: can an agent deliver a usable file for a long office task, and how much human labor does that file represent [^1]?
What happened
The benchmark comes from the Agent Frontier Team at Baidu and contains 100 tasks adapted from real office requests that practitioners proposed [^1]. The team started with 1,715 proposals, narrowed them to 595 and then to 282 candidates, and picked the final 100. Instructions were rewritten to remove identifying details [^1]. The tasks come with 220 input files, including images, videos, Word documents, slide decks, spreadsheets and PDFs, and ask for finished output files [^1]. Annotators did every task without AI help. The average task took 2.32 hours of human labor, the median 2.03 hours and the longest 8.35 hours [^1]. Each task also has a price proxy: for about 20 percent of tasks it comes from what practitioners had paid to outsource similar work, for the rest from three experts' estimates. The mean is $6.86 [^1].
Scoring is done by code rather than by human reviewers or an LLM judge, which the authors say keeps scores comparable as judge models change [^1]. A file that fails any usability check, for example because it can't be opened or edited, scores zero. Otherwise rubric items award points for completed requirements and subtract points for unintended damage the user would have to repair [^1]. The benchmark has 219 usability checks and 2,009 task-completion items [^1]. All models ran in the same in-house scaffold with shell, file and scripting tools but no GUI control, under a time limit of 14,400 seconds per task [^1].
GLM-5.2 led the models with 17.91, followed by Qwen3.7-Plus at 17.51, Kimi K2.6 at 17.00, DeepSeek-V4-Pro at 14.48 and MiniMax M3 at 13.82 [^1]. The human baseline, built from the best annotator submission for each task, reached 27.79 [^1]. Qwen3.7-Plus averaged $0.2152 in token cost and 0.193 hours per task, GLM-5.2 $1.4823 and 0.521 hours. For humans, the paper lists the price proxy of $6.8560 and 2.324 hours [^1].
Why it matters
The economic labels change the ranking. Weighted by human labor time or by price, Qwen3.7-Plus scored highest among the models, not GLM-5.2, which the authors read as relatively better results on the longer or pricier tasks [^1]. Their conclusion is that the model with the best average is not necessarily the one that handles the most economically important work [^1]. Scores also fell as human labor time rose, for people and models alike, and more steeply for the models. The authors take that as a sign that agents struggle to keep track of task state and quality over long workflows [^1].
Cheap output is only worth something if it's usable. Humans scored zero on 29 percent of tasks; Qwen3.7-Plus had the lowest zero-score rate among the models at 38 percent, and DeepSeek-V4-Pro and MiniMax M3 reached 50 and 51 percent [^1]. The cost comparison also has limits the paper states itself: human cost is the price proxy, model cost is the token bill, and runtime depends on the scaffold, the environment and API latency, so the authors treat it as a reference rather than an exact figure [^1]. The tests cover only programmatic file work, not agents operating office apps through the screen [^1]. The full benchmark, including tasks, rubrics, verifier code and economic labels, is released [^1].
Practical example
Say you lead a small team and want an agent to build the monthly report deck from a spreadsheet and a few Word notes. This benchmark suggests weighing two numbers. The token bill is likely small next to the price of the job: in the paper, Qwen3.7-Plus averaged $0.2152 per task, while the tasks' price proxy averaged $6.86 [^1]. But even that model handed back a zero-score result on 38 percent of tasks [^1]. So before the deck goes out, you check what the benchmark's verifiers check: whether the file opens and can be edited, whether it meets each requirement you gave, and whether anything you didn't ask for was changed [^1].
Related gear
We recommend this book because it looks at how businesses redesign work when people and AI systems share the tasks, the trade-off this benchmark puts numbers on.