Back to AI News
arxiv

AutoBuddy-Bench: Evaluating Proactive Always-On Agents on Task Discovery and Execution

The paper introduces AutoBuddy-Bench, a simulator for proactive task discovery and execution, where an agent must infer a user's pending obligations from workspace artefacts and complete them in the systems where the work lives. The simulator models an organisation's employees, projects, business systems and event history as a temporal graph, renders it into realistic mail, chat, documents and records, derives ground-truth tasks from obligation-creating events, and judges execution by the agent's changes to the workspace. The authors also build AutoBuddy, a reference agent with interchangeable backbone LLMs that grounds each discovered task in its source records and executes it in a separate session with write access. Varying only the backbone, they report a systematic shift in where tasks are lost: weaker models fail mainly at discovery, intermediate models identify tasks but often stop short of executing them, and the strongest models fail mainly during execution, most often on business-system operations, while discovery precision is only weakly related to execution quality.

AI Brief

It offers a concrete benchmark and reference agent for evaluating whether always-on LLM agents can find and complete work unprompted, an under-tested capability.

arXiv:2611.12393v1 Announce Type: new Abstract: As LLM agents become more autonomous, they are increasingly expected to run as always-on assistants on a user's behalf, identifying work to be done and completing it without being asked. Yet existing benchmarks usually give the agent its task, whereas in knowledge work the requests sit in the user's workspace and the agent must first recognise which obligations exist. We study proactive task discovery and execution, in which an agent must infer a user's pending tasks from their workspace and complete them in the systems where the work lives. This setting poses two challenges. For the agent, the tasks are buried in a large volume of mail, chat, documents and records. For evaluation, a benchmark needs a simulated organisation whose people, events and documents are mutually consistent, so that tasks arise realistically and their correct outcomes are known. We introduce AutoBuddy-Bench, a simulator that addresses these challenges. It models an organisation's employees, projects, business systems and event history as a temporal graph, renders it into realistic workspace artefacts, derives ground-truth tasks from the events that create obligations, and computes each task's outcome from the world state; execution is judged by the agent's changes to the workspace. To evaluate current LLMs, we build AutoBuddy, a reference agent with interchangeable backbone LLMs: it identifies tasks from the workspace, grounding each in its source records, and executes each in a separate session with write access. Varying only the backbone, we find a systematic shift in where tasks are lost: weaker models fail mainly at discovery, intermediate models identify tasks but often stop short of carrying them out, and the strongest models fail mainly during execution, most often on business-system operations. Discovery precision is only weakly related to execution quality, suggesting the two are distinct capabilities.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!