Live intelligence

Latest AI updates

4 updates
Research

WebArena-Pro: A Reproducible Multimodal Benchmark for Web Agents

WebArena-Pro is a new benchmark for web agents comprising 300 tasks across 20 self-hosted web applications in six domain categories. It extends evaluation beyond text and vision by treating audio and video as core observations, with dedicated actions for extracting information from each. The abstract reports top performance of 62.3% from Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5, while the strongest GenericAgent configurations reached 30.3%. The benchmark runs tasks in isolation to support reproducible parallel evaluation and aims to combine live-benchmark coverage with human-validated reproducibility.

WebArena-Proweb agentsmultimodal evaluation
Research

Scale-Induced Semantic Shift Enables Visual Prompt Injection in VLMs

Researchers describe Scale-Induced Semantic Shift (SISS), a security failure mode in which image resizing or compression changes machine-readable semantics and makes hidden textual instructions recoverable after aggressive downsampling. Their closed-loop black-box framework reaches a 58.75% average attack success rate on a fixed 300-case benchmark across four closed-source vision-language models, though the effect is bounded to aggressive compression. In a downstream web-agent study, hidden-instruction perception increases but absolute behavioral compromise remains about 4%. The proposed ScaleGuard consistency detector reduces attack success, while generic input transformations provide little protection.

Visual Prompt InjectionVision-Language ModelsAI Security
Research

ANGLER and REEF: Adaptive Red-Teaming Benchmark for Web Agents

Researchers introduce REEF, a benchmark of red-teaming executable environments with state-based verifiers for measuring web agent security, covering 9 websites across travel, commerce, productivity, media and communication. Each benign task is paired with a malicious objective delivered through 5 attack types at the user and environment level. ANGLER builds on REEF's execution feedback to evolve attacks via LLM-based mutation and near-miss seed selection rather than a fixed attack set. Across 8 VLM-based web agents, ANGLER reached a 20.36% end-to-end attack success rate versus 15.24% for the non-adaptive setting, and cut benign task success from 61.9% to 44.4%.

web agentsAI securityred-teaming
Research Editor selected

DynaRev Benchmarks Web Agents on Changing User Goals

DynaRev introduces 181 tasks that evaluate how web agents respond when users revise requirements during execution, measuring correctness, responsiveness and efficiency. Evaluations across five models show that agents often fulfill only part of the requirements, with results varying by task source, revision type and model. Further analysis finds gains from rewriting instructions while preserving browser state, but losses from browser resets; code and data are available.

Why it matters: The benchmark tests agents’ ability to handle evolving user requirements, exposing limitations that fixed-goal evaluations may miss.

DynaRevWeb agentsAgent evaluation