TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs
TwinLab-Bench is a new benchmark that evaluates AI agents acting as "AI scientists" on experiments they design themselves, covering 52 tasks drawn from physical-science, life-science and engineering laboratories and supporting more than 100 experimental action types. Each task pairs the tested agent with its own TwinLab agent that serves as the environment, executing the designed experiments by reading laboratory raw materials and calling calculators built from lab records, with reports scored against hidden answers taken from real laboratory records. The authors report that the TwinLab agent largely eliminates readings that deviate from real records and inconsistent answers to repeated requests, while closed-book probes control for answers recalled from memory. The benchmark remains hard for frontier models: the strongest, Claude Fable 5.2, averages only 45.0, and the paper includes an empirical study of where it fails.
It offers a concrete, record-grounded benchmark and failure analysis for autonomous experiment-design agents, a capability earlier benchmarks largely skipped.
Comments & Questions (0)
No comments yet
Be the first to comment!