Back to AI News
arxiv

Mind2Act World benchmark separates robotic reasoning, action, and their composition

Researchers introduced Mind2Act World, a diagnostic benchmark for embodied robotic manipulation with 30 simulated tasks across five real-world application domains. Each task has three difficulty levels and Clean or Random scene settings, producing 180 configurations. The benchmark evaluates systems through separate Mind, Act, and closed-loop Mind2Act views, and includes a harness that attributes failed episodes to reasoning, execution, or their composition. In evaluations of multimodal agents, vision-language-action models, and world-action models, the strongest agent reached 52.7% on Mind and 39.0% on Act episodes but only 12.0% on Mind2Act episodes.

AI Brief

It provides a structured way to diagnose whether robotic failures stem from reasoning, physical execution, or the difficulty of composing them through environmental feedback.

arXiv:2611.26444v1 Announce Type: cross Abstract: Embodied intelligence aims to let robots perform complex physical tasks with human-like adaptability and reliability, which requires three complementary capabilities: history-based cognitive reasoning, precise and timely physical execution, and the composition of reasoning and execution through environmental feedback. Yet existing benchmarks either evaluate reasoning and execution within the same tasks or assess them separately without their composition, so they reveal little about whether a failure comes from reasoning, execution, or their composition. We introduce Mind2Act World, a diagnostic benchmark of 30 simulated manipulation tasks from five real-world application domains. Each task has three difficulty levels and two scene settings (Clean and Random), yielding 180 configurations. The tasks form three views: Mind tests decisions based on past observations, Act tests precise and timely execution, and Mind2Act combines both in closed-loop tasks where action outcomes guide later decisions. Comparing the three views shows whether a system lacks a capability or fails to compose capabilities it already has. A diagnosis harness further attributes the failed episodes of three representative systems to Mind, Act, or their composition and summarizes each system in a profile card. Our evaluation of multimodal agents, vision-language-action models (VLAs), and world-action models (WAMs) shows that competence in a single view does not carry over to Mind2Act tasks. Even the strongest agent succeeds in 52.7% of Mind and 39.0% of Act episodes but in only 12.0% of Mind2Act episodes. The strongest agent retains task state but struggles with sustained contact, whereas end-to-end policies execute practiced motions but retain little task state. These findings point to composing reasoning and execution through environmental feedback as the main open problem.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!