Back to AI News
arxiv

STEMBench scores LLMs on postdoc-level physics, astrophysics and neuroscience problems

Researchers introduced STEMBench, a benchmark of twelve research problems in high-energy physics, astrophysics and neuroscience that are graded by each field's own figure of merit rather than by answer keys or unit tests. Each problem ships with two measured anchors — a no-domain-knowledge submission and an expert reference reproducing a published method — both produced by running the benchmark's released code. Across eight frontier models, task-level wins and domain-level leadership diverged: one model won five of twelve tasks yet led no domain mean. In a high-effort comparison, at least one model's mean score exceeded the expert anchor on seven of twelve tasks, while the other five involved uncertainty estimation, latent and causal inference, and cross-site clinical classification.

AI Brief

It offers a field-grounded way to measure whether LLMs contribute genuine scientific insight, with released tasks, graders, anchors and harness for reproducible evaluation.

arXiv:2611.24321v1 Announce Type: new Abstract: Scientific problems have no answer key, so evaluating an agent means deciding what counts as a better solution before any agent is run. Existing benchmarks either match a symbolic answer against a key or, where they require executable work, score it by unit tests, by a rubric, or by position on a competition leaderboard. None of these is the criterion by which a field decides that one published method beats another. STEMBench adopts that criterion directly. Each of its twelve research problems in high-energy physics, astrophysics and neuroscience is scored by its own field's figure of merit, and each comes with two measured anchors: a submission that uses no domain knowledge, and an expert reference reproducing a published method, both obtained by running code shipped with the benchmark. The interval between them states how much of a problem is crossed by scientific insight rather than by engineering. Across eight frontier models, task-level wins and domain-level leadership dissociate: a model that wins five of the twelve tasks leads no domain mean. In the high-effort cross-model comparison, at least one model's mean score exceeds the expert anchor on seven of twelve tasks. The remaining five span uncertainty estimation, latent and causal inference, and cross-site clinical classification. Numerical scores and official-submission validity capture different aspects of performance and are reported separately. We release the tasks, graders, anchors and harness.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!