Back to AI News
arxiv

WebArena-Pro: A Reproducible Multimodal Benchmark for Web Agents

WebArena-Pro is a new benchmark for web agents comprising 300 tasks across 20 self-hosted web applications in six domain categories. It extends evaluation beyond text and vision by treating audio and video as core observations, with dedicated actions for extracting information from each. The abstract reports top performance of 62.3% from Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5, while the strongest GenericAgent configurations reached 30.3%. The benchmark runs tasks in isolation to support reproducible parallel evaluation and aims to combine live-benchmark coverage with human-validated reproducibility.

AI Brief

It gives agent developers a broader, human-validated and multimodal benchmark for comparing web-agent performance across many self-hosted applications.

arXiv:2611.17955v1 Announce Type: new Abstract: Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by both Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, followed by OpenCode with Kimi 3.1 Pro at 60.7%. The strongest GenericAgent configurations, Gemini 3.1 Pro omni and Gemma4 31B, both achieve 30.3%. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release. Project page: https://webarena-pro.github.io
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!