Back to AI News
arxiv

TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs

TwinLab-Bench is a new benchmark that evaluates AI agents acting as "AI scientists" on experiments they design themselves, covering 52 tasks drawn from physical-science, life-science and engineering laboratories and supporting more than 100 experimental action types. Each task pairs the tested agent with its own TwinLab agent that serves as the environment, executing the designed experiments by reading laboratory raw materials and calling calculators built from lab records, with reports scored against hidden answers taken from real laboratory records. The authors report that the TwinLab agent largely eliminates readings that deviate from real records and inconsistent answers to repeated requests, while closed-book probes control for answers recalled from memory. The benchmark remains hard for frontier models: the strongest, Claude Fable 5.2, averages only 45.0, and the paper includes an empirical study of where it fails.

AI Brief

It offers a concrete, record-grounded benchmark and failure analysis for autonomous experiment-design agents, a capability earlier benchmarks largely skipped.

arXiv:2611.26248v1 Announce Type: new Abstract: AI agents are increasingly expected to act as AI scientists, which requires three increasingly open abilities: carrying out a given computational workflow correctly, choosing the next measurements from a given design space, and designing their own experiments. Existing benchmarks mainly evaluate the first two abilities, in a terminal or in a closed loop that selects from predefined candidates. Evaluating the third ability requires reliably executing an experiment that an agent designs, with readings grounded in a real laboratory. We introduce TwinLab-Bench, which evaluates AI scientists on self-designed experiments in 52 tasks from laboratories in the physical sciences, the life sciences, and engineering. The benchmark supports more than 100 experimental action types, and the agent's report is scored by comparison with a hidden answer taken from real laboratory records. Each task has its own TwinLab agent, an agent that serves as the environment and executes the experiments that the tested agent designs, reading the raw materials of the laboratory and calling calculators that are built from its records. Compared with a language model that sees only the task description, the TwinLab agent almost eliminates readings that deviate from the real records and inconsistent answers to a repeated request, and closed-book probes control for answers recalled from memory. We find that these tasks remain hard for frontier models: the strongest, Claude Fable 5.2, averages a score of only 45.0, and we conduct an empirical study of where it fails.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!