Benchmark Audits LLM Peer Reviewers on Real Scientific Papers
Researchers introduce a benchmark that injects objective, localizable errors of four types at three severity levels into real papers and measures reviewer detection by error type and severity, along with false-positive rates on clean copies. They evaluate twelve reviewer systems, including six single-LLM reviewers, two Claude Code configurations, three public reviewers, and a multi-agent reviewer called Skepthical, across astrophysics and machine-learning corpora of 15 recent arXiv preprints each with 357 total injected errors. Model choice had the largest impact on detection, with three frontier LLMs differing by about 50 percentage points under the same prompt. Tooling and scaffolding helped when they supplied capabilities the model lacked, such as enforced citation retrieval improving citation-error detection on astrophysics but not on ML. False-positive rates were below 10% for most systems but varied with prompt, model, and scaffolding.