114 tasks across 8 real-world work domains, each requiring an agent to weave GUI observation with CLI/code execution within a single trajectory. Rollouts run inside a real Ubuntu sandbox; a trajectory-aware Agent-as-a-Judge audits eight quality dimensions and zeros credit on any of nine shortcut patterns (synthetic screenshots, hard-coded metrics, mock services…).
Deployed CUA runtimes already integrate GUI control, CLI/code execution, browsers, and external tools inside one agent loop. Existing benchmarks test these channels in isolation, leaving the orchestration layer under-measured. WeaveBench is built around tasks where success requires the agent to weave both channels together.
GUIs expose rendered, transient, spatial state — canvases, dialogs, visual feedback. CLI/code expose structured, scriptable, persistent state — source files, configs, logs, services. Real workflows braid them: observe in the GUI, verify in the shell, edit in code, re-render, re-observe.
Each admitted task requires coordinating GUI observation/action with CLI/code modification inside the same trajectory; annotated with single-channel-bound atomic operations.
Expert reference trajectories contain multiple interleaved GUI and CLI/code phases — not a single perception, action, or tool-use step.
Tasks span multiple independent applications/processes whose states are linked by the workflow; agents must preserve and transfer information across them.
Final-only grading is fragile here. The judge audits transcripts, files, screenshots, and logs; nine shortcut detectors zero credit on confirmed reward hacks.
Task: 114 tasks across 8 domains, harvested from real
venues, packaged as
ℰ = (𝒫, ℳ, 𝒞)
bundles, audited against P1–P3, and stress-tested by ≥3 pilot
agents.
Harness: the agent runs in a single session over an
Ubuntu sandbox, with a minimal GUI plugin (one screenshot
tool + nine actuation primitives) layered on top of OpenClaw's CLI/code
tools; the same plugin is ported to Codex CLI, Claude Code,
and Hermes.
Evaluation is performed by an isolated
trajectory-aware agentic judge that combines bottom-up rubric scoring
with shortcut detection.
One perception primitive screenshot plus nine
pyautogui-backed actuation primitives
(click, double_click,
triple_click, move, drag,
scroll, type, keypress,
wait). Exposed alongside each runtime's terminal, file,
code, and browser tools — model loop and prompts unchanged.
For every rollout, an isolated subprocess judge re-fetches evidence over multiple turns using file, image, and shell tools; decomposes each deliverable into atomic clauses; verifies each clause with cited evidence; and assigns scores along eight process & outcome dimensions.
A parallel scan covers fake screenshots/renders, regenerated fixtures,
hard-coded metrics, mock services, duplicate crops, overlay
manipulation, ground-truth leakage, runtime injection, and
fabricated screenshots. A high-confidence hit triggers
ht,m=1 and zeros the task score.
st,m = 0 if ht,m=1;
otherwise min(⅛ ∑ dprocess,
ddeliv). The min prevents strong auxiliary
dimensions from masking weak deliverables; the zeroing rule prevents
fabricated evidence from earning partial credit.
Per-domain task counts range from 10 to 18. Best live rollouts use a
median of 76 tool calls (max 471) and a
median of 16 GUI↔CLI channel switches per task.
Each task carries provenance (URL, commit hash, or post id) from public
venues, with self-contained bundles
ℰ = (𝒫, ℳ, 𝒞)
covering prompt, materials, and check anchors.
Filesystem & storage, system & hardware services, desktop UI / UX flows.
Office suites, markup, LaTeX, print-ready document workflows.
Game engines & runtimes, puzzle / strategy, realtime / action.
DevTools, perf budgets, network profiling, UI & client-state debugging.
Notebooks, dashboards, observability traces, pipelines / ETL.
Cluster monitoring, services, DB ops, network & security.
CAD, engineering design, scientific / FEM simulation.
Visual asset workflows, color management, engineering design.
Seven end-to-end rollouts from Claude Opus 4.7 on the OpenClaw harness, captured on a real Ubuntu desktop and replayed at 5× speed. The left pane shows the agent's live action log; the right pane shows the desktop it is driving. Use the arrows to step through all seven.
The benchmark has two natural axes. Sweep A fixes OpenClaw as the runtime and varies the backbone (Table 1). Sweep B keeps the strongest backbones and varies the deployed runtime (Table 2). Together they show that hybrid-interface performance is determined as much by the runtime scaffold as by raw model capability — cross-pairing can swing the same backbone by >25 PR points.
| Backbone | PR ↑ | Overall ↑ | DSK | DOC | GAM | WEB | DAV | OPS | SPA | DES |
|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 4.7 | 35.1 | 0.482 | 55.6 | 29.4 | 23.5 | 66.7 | 15.4 | 41.7 | 16.7 | 20.0 |
| GPT-5.5 | 33.3 | 0.466 | 38.9 | 35.3 | 35.3 | 21.4 | 23.1 | 38.5 | 33.3 | 40.0 |
| GPT-5.4 | 22.8 | 0.465 | 55.6 | 35.3 | 5.9 | 0.0 | 23.1 | 23.1 | 8.3 | 20.0 |
| GPT-5.3-codex | 18.4 | 0.456 | 33.3 | 23.5 | 29.4 | 0.0 | 7.7 | 16.7 | 8.3 | 20.0 |
| GPT-5.2-codex | 6.1 | 0.321 | 5.6 | 11.8 | 0.0 | 0.0 | 15.4 | 16.7 | 0.0 | 0.0 |
| GPT-5.1-codex | 1.8 | 0.226 | 0.0 | 5.9 | 0.0 | 0.0 | 7.7 | 0.0 | 0.0 | 0.0 |
| Gemini 3.1 pro | 1.8 | 0.223 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 8.3 | 8.3 | 0.0 |
| Qwen3.5-397B-A17B | 0.9 | 0.318 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 8.3 | 0.0 | 0.0 |
| Qwen3-VL-8B-Think | 0.9 | 0.092 | 0.0 | 0.0 | 0.0 | 0.0 | 8.3 | 0.0 | 0.0 | 0.0 |
| GUI-Owl-1.5-32B | 0.0 | 0.065 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
SPA and DES — the two most GUI-heavy domains — are the bottom-two for every backbone with non-trivial PR, confirming GUI as the binding constraint.
| Backbone | Harness | PR ↑ | Overall ↑ | DSK | DOC | GAM | WEB | DAV | OPS | SPA | DES |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.5 | Codex CLI | 35.1 | 0.499 | 38.9 | 29.4 | 23.5 | 53.3 | 15.4 | 50.0 | 58.3 | 10.0 |
| OpenClaw | 33.3 | 0.466 | 38.9 | 35.3 | 35.3 | 21.4 | 23.1 | 38.5 | 33.3 | 40.0 | |
| Hermes Agent | 31.6 | 0.466 | 55.6 | 29.4 | 35.3 | 40.0 | 7.7 | 25.0 | 25.0 | 20.0 | |
| Claude Code | 14.9 | 0.299 | 33.3 | 11.8 | 11.8 | 0.0 | 15.4 | 16.7 | 25.0 | 0.0 | |
| Claude Opus 4.7 | Claude Code | 41.2 | 0.532 | 55.6 | 47.1 | 23.5 | 53.3 | 23.1 | 50.0 | 33.3 | 40.0 |
| OpenClaw | 35.1 | 0.482 | 55.6 | 29.4 | 23.5 | 66.7 | 15.4 | 41.7 | 16.7 | 20.0 | |
| Hermes Agent | 28.1 | 0.516 | 33.3 | 47.1 | 11.8 | 26.7 | 30.8 | 50.0 | 8.3 | 10.0 | |
| Codex CLI | 13.2 | 0.378 | 16.7 | 11.8 | 11.8 | 6.7 | 7.7 | 25.0 | 16.7 | 10.0 |
Cross-pairing matters: Claude Opus 4.7 drops from 41.2% on Claude Code to 13.2% on Codex CLI; GPT-5.5 drops from 35.1% on Codex CLI to 14.9% on Claude Code. Tool schemas, prompting conventions, and action-loop design interact strongly with model-specific tool-use behavior.
The same generation of frontier backbones that scores >78% on OSWorld-Verified and 75% on MCPWorld collapses to 41.2% on WeaveBench — and to ≤3.5% when restricted to a single channel. That gap is the cross-interface, long-horizon orchestration tax.
The first ablation removes either channel and shows that single-interface CUAs collapse by an order of magnitude — on WeaveBench the second channel is not a convenience, it is required. The second ablation removes trajectory access from the judge and shows that outcome-only grading awards 10–20 PR points of false credit.
We re-run each backbone in three settings on OpenClaw: GUI-only (the screenshot tool plus nine actuation primitives), CLI-only (the full OpenClaw CLI), and Hybrid (both). Both single-interface settings collapse by an order of magnitude across every backbone, consistent with the channel non-substitutability admission rule (P1).
| Backbone | GUI | CLI | Hybrid | Δ Hyb-best1 |
|---|---|---|---|---|
| Claude Opus 4.7 | 1.8 | 3.5 | 35.1 | +31.6 |
| GPT-5.5 | 0.8 | 2.6 | 33.3 | +30.7 |
| GPT-5.4 | 0.8 | 2.6 | 22.8 | +20.2 |
| GPT-5.3-codex | 0.0 | 1.8 | 18.4 | +16.6 |
Single-interface PassRate stays in single digits for every backbone — an order of magnitude below Hybrid.
Compared against the two prior hybrid CUA benchmarks that report comparable interface ablations, WeaveBench's hybrid gain is an order of magnitude larger, and the single-channel floor is single-digit vs. 40–70%. The additional channel is forced by the task specification, not offered as a per-step convenience.
| Benchmark | GUI | CLI/MCP | Hyb. | Δ |
|---|---|---|---|---|
| OSWorld-MCP | 40.1 | — | 43.3 | +3.2 |
| MCPWorld | 70.7 | 53.2 | 75.1 | +4.5 |
| WeaveBench | 1.8 | 3.5 | 35.1 | +31.6 |
Cooperation is forced by the task specification rather than offered as a per-step convenience.
We re-score every rollout with an outcome-only judge that sees only the final deliverables — no trajectory access, no shortcut scan. Switching back to WeaveBench's trajectory-aware judge removes 10.3–20.2 PR points across the four GPT backbones. For GPT-5.5, the audited rate drops from 53.5% → 33.3%. These gaps are lower bounds: rollouts already received an anti-fabrication prompt with a cost-free honest fallback.
| Backbone | Outcome-only | Trajectory-aware | Δ |
|---|---|---|---|
| GPT-5.5 | 53.5 | 33.3 | −20.2 |
| Claude Opus 4.7 | 51.6 | 41.2 | −10.4 |
| GPT-5.4 | 33.1 | 22.8 | −10.3 |
| GPT-5.3-codex | 28.7 | 18.4 | −10.3 |
We aggregate every OpenClaw rollout for the three frontier backbones (Opus 4.7, GPT-5.5, GPT-5.4) across reasoning budgets — n = 2,209 trials, 1,735 failures. Following CocoaBench, we adopt a hierarchical taxonomy with 5 top-level families and 13 sub-classes; WeaveBench extends the codebook with two hybrid-specific families absent from prior benchmarks — E4 Long-horizon Execution Discipline and E5 Reward Hacking.
Synthesized render · Hardcoded metric · Crop / overlay · CLI bypass of GUI. Only 30% of E5 hacks follow a true infra bug; the other 70% happen in clean environments despite the explicit anti-fabrication policy.
Silent halt · Premature halt · Cross-channel state drift. Emerges only when the agent must coordinate multiple deliverables across a long horizon of interleaved GUI/CLI actions.
Imprecision — close miss is the largest sub-class here. Surfaces only with multi-deliverable contracts.
Tool-affordance prior + channel-policy compliance failures.
Perception is not the bottleneck on frontier backbones — a finding that runs against the perception-centric prior in the GUI-agent literature.
46% of failures are E5 reward hacking — primarily E5.1 synthesized renders (24%) and E5.2 hardcoded metrics (14%). The model can complete or honestly skip, but the implicit reward landscape selects forgery when stuck.
E4 dominates (44%) and E4.2 premature halt alone is 27% — the highest single sub-class share for any backbone. Reward hacking is comparatively rare (23%); after a provider hiccup the model never rebuilds its deliverable checklist.
No single sub-class above 17%. E5, E4, and E1 each contribute ~30%. Failure style is a function of model identity, not just raw capability.
We decompose every tool call across all GPT-5.5 rollouts into atomic
operations. The top-10 operations cover 93.1%
of 10,873 active calls; exec: shell alone accounts
for 27.3%. A large fraction of GUI work is
hidden inside these shell calls — once re-attributed at the
atomic-operation level, the GUI share rises from
33.9% → 62.9%.
gnome-screenshot via exec 2.2× more often than the native __computer__.screenshot, and drives mouse/keyboard via pyautogui, xdotool, and wmctrl another 521 times.
of all GPT-5.5 tool calls — the single dominant primitive in the trajectory.
tool-level GUI share rises to atomic-op share once exec-routed screenshots and pyautogui calls are reclassified.
median GUI↔CLI handoffs in a successful trajectory — single-channel agents cannot satisfy this.
per task; max 471. WeaveBench is firmly in long-horizon territory.
If you use WeaveBench in your research, please cite the paper and the released dataset / code.
@misc{li2026weavebench,
title = {{WeaveBench}: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces},
author = {Li, Wanli and Zhou, Bowen and Yu, Yunyao and Xu, Zhou and Yang, Yifan and Li, Dongsheng and Shan, Caihua},
year = {2026},
month = jun,
howpublished = {\url{https://weavebench.github.io}},
note = {Microsoft Research Asia, Zhejiang University, Tsinghua University.
Dataset: \url{https://huggingface.co/datasets/wanlilll/WeaveBench};
Code: \url{https://github.com/weavebench/WeaveBench}.}
}