Back to AI News
arxiv

Embodied First Exam benchmark tests coding agents on robot control

The paper introduces Embodied First Exam, a unified agent-native benchmark for coding agents that control robots. It covers 1,156 tasks across 35 suites and 47 robots, including new suites for spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Agents can choose direct control, code-to-control, or policy training, with Standard, Realtime, and Privileged evaluation settings. On 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, reaches a 34% success rate in the Standard Setting.

AI Brief

It offers a concrete, broad benchmark and baseline results for assessing whether coding agents can reliably operate robots under physical-world constraints.

arXiv:2611.12177v1 Announce Type: cross Abstract: Early successes on agentic robotics point to a potential paradigm shift toward using coding agents to control robots. Existing robotics benchmarks are built around visuomotor policy learning, leaving it open whether coding agents can reliably control diverse robots under physical-world constraints. We therefore introduce the Embodied First Exam, a unified, comprehensive, and agent-native evaluation framework where coding agents can choose their own control approach: direct control, code to control, or train a policy to control. We hope it serves as a first exam for coding agents before they enter the physical world, with 1,156 tasks in 35 suites on 47 robots, including 9 new suites that go beyond pick-and-place to spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Each task can be run in up to three evaluation settings. The Standard Setting mimics the real world: the agent receives only sensor observations and, by default, cannot reset the environment. Its Realtime variant keeps the environment running while the agent reasons. The Privileged Setting gives the agent access to the full simulator state and unlimited resets, allowing it to search for an oracle solution for training purposes. Success rates in this setting serve as an approximate upper bound. On the 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, achieves a success rate of 34% in the Standard Setting, leaving much room for improvement. To see where agents fail, we label the stage at which failed attempts go wrong and compare failures across agents, settings, and stage categories.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!