arXiv:2611.06910v1 Announce Type: new
Abstract: AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
arxiv
FrontierPhysics: A New Benchmark for End-to-End AI Physics Research
FrontierPhysics is a benchmark created by 42 physicists with 56 tasks spanning theoretical, computational, and experimental physics. It evaluates AI agents on the full research pipeline—literature review, planning, experimentation, and manuscript writing—using both deterministic verifiers and expert rubrics. Across 22 models in 26 configurations, no agent passed more than 46 of 168 rollouts, with the main weaknesses being insufficiently rigorous reasoning and limited expert intuition rather than knowledge gaps.
Provides a rigorous, expert-curated evaluation of frontier AI agents' ability to conduct end-to-end physics research, highlighting critical reasoning deficiencies.
Comments & Questions (0)
No comments yet
Be the first to comment!