ClawBench

ClawBench is an open benchmark and public leaderboard that measures whether AI browser agents can complete everyday online tasks on real, live websites. Agents receive instructions such as ordering a specific meal with a delivery note, booking travel or applying for a job, and drive a real browser on the actual site. The current corpus covers 130 tasks across 63 live platforms. It is aimed at model developers, agent framework builders and researchers who need evidence of real world agent performance rather than results from simulated sandboxes. Scoring runs in two stages. The harness first intercepts the agent's final submit request before it is sent and checks deterministically whether the URL and method match the task schema; a judge model then reads the intercepted payload against the instruction under both a lenient and a strict rubric. The leaderboard reports reward, interception rate, strict reward, cost per task and pass counts for each model, split by harness, including Hermes, OpenClaw, Codex and Claude Code runs, across the first and second versions of the corpus. Anyone can evaluate their own model by installing the evaluation package from PyPI and submitting run artifacts through a pull request on GitHub, and an accompanying arXiv paper documents the method. ClawBench is free to use, and its code is released under the Apache 2.0 license. Running the benchmark incurs whatever model and compute costs the evaluator's chosen provider charges; the project reports spending more than $10,000 in frontier model compute to produce its own trace corpus. Because tasks run against live sites that change over time, scores from different snapshot dates are not strictly comparable, and the leaderboard states the snapshot date of its published results.