arXiv:2610.40133v1 Announce Type: new
Abstract: Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. DynaRev uses semantic milestones to deliver revisions at task-defined stages, covers additive revisions, unconditional overrides, and runtime-conditioned overrides, and evaluates correctness, responsiveness, and efficiency through trajectory-grounded evaluation. Five-model evaluations show that agents often satisfy only part of the task requirements, with performance varying across task sources, revision types, and models. Further analyses reveal varied milestone timing, mixed effects of dynamic delivery, and contrasting revision-handling outcomes, with gains from instruction rewriting while preserving browser state and losses from browser resets. DynaRev provides a testbed for studying how web agents adapt to evolving user goals. The code and data are available at https://github.com/zsq259/DynaRev.
arxiv
DynaRev Benchmarks Web Agents on Changing User Goals
DynaRev introduces 181 tasks that evaluate how web agents respond when users revise requirements during execution, measuring correctness, responsiveness and efficiency. Evaluations across five models show that agents often fulfill only part of the requirements, with results varying by task source, revision type and model. Further analysis finds gains from rewriting instructions while preserving browser state, but losses from browser resets; code and data are available.
The benchmark tests agents’ ability to handle evolving user requirements, exposing limitations that fixed-goal evaluations may miss.
Comments & Questions (0)
No comments yet
Be the first to comment!