AutoBuddy-Bench: Evaluating Proactive Always-On Agents on Task Discovery and Execution
The paper introduces AutoBuddy-Bench, a simulator for proactive task discovery and execution, where an agent must infer a user's pending obligations from workspace artefacts and complete them in the systems where the work lives. The simulator models an organisation's employees, projects, business systems and event history as a temporal graph, renders it into realistic mail, chat, documents and records, derives ground-truth tasks from obligation-creating events, and judges execution by the agent's changes to the workspace. The authors also build AutoBuddy, a reference agent with interchangeable backbone LLMs that grounds each discovered task in its source records and executes it in a separate session with write access. Varying only the backbone, they report a systematic shift in where tasks are lost: weaker models fail mainly at discovery, intermediate models identify tasks but often stop short of executing them, and the strongest models fail mainly during execution, most often on business-system operations, while discovery precision is only weakly related to execution quality.
It offers a concrete benchmark and reference agent for evaluating whether always-on LLM agents can find and complete work unprompted, an under-tested capability.
Comments & Questions (0)
No comments yet
Be the first to comment!