Back to AI News
arxiv

Joi-Bench-Task benchmarks task coordination in full-duplex voice agents

The paper introduces Joi-Bench-Task, a benchmark designed to isolate coordination between task execution and conversation in full-duplex voice agents. It uses an interactive user simulator that injects configurable interventions during tool execution and spoken result delivery for reproducible evaluation. Five commercial voice agents are assessed on execution responsiveness, coordination under request changes, and coordination under interruptions. The reported results show a maximum 55% result completion rate under request changes and average information omission rates of at least 23% after interruption for four of the five agents.

AI Brief

It provides a reproducible benchmark and empirical evidence on task-coordination failures that matter for builders and evaluators of voice agents.

arXiv:2611.12118v1 Announce Type: new Abstract: Full-duplex voice agents enable natural, hands-free interaction, allowing users to refine requests, redirect ongoing work, and seek clarification while the agent is speaking. However, existing voice-agent benchmarks insufficiently isolate coordination between task execution and conversation. We introduce a benchmark that assesses this coordination by examining whether agents track what is being executed, what has been completed, what has been communicated, and what remains to be reported as the conversation evolves. We propose an interactive user simulator that introduces configurable interventions at designated timing in tool execution and spoken result delivery, supporting controllable and reproducible evaluation conditions. We evaluate five leading commercial voice agents along three dimensions: execution responsiveness, coordination under request changes, and coordination under interruptions. Agent evaluations reveal persistent coordination failures. Under request changes, the highest result completion rate is only 55%, indicating difficulty for current voice agents adapting to new requests while preserving earlier obligations. After user interruption, four of the five agents exhibit average information omission rates of at least 23%. These findings highlight task coordination as a central challenge for current voice agents in complex, evolving conversations.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!