Back to AI News
arxiv

Gandalf: An Open-Source Verifier for Rubric-Graded Agent Environments

This arXiv paper presents the first systematic study of verifiers used in rubric-graded agent environments, where a verifier judges natural-language criteria against agent deliverables and environment state changes. The authors introduce BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks, and derive a nine-capability taxonomy of verifier requirements distilled into three design principles: reactive verification, environment alignment, and domain guidance. Implemented as the open-source verifier Gandalf, it matches or exceeds three existing verifiers on BVB at lower cost — with Gemini 3 Flash it reaches F1 0.864 versus 0.852 for the strongest baseline at one-fifth the cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, achieving the highest F1 there (0.955 vs. 0.934 for the best baseline), and the authors release BVB, Gandalf, and reproduction code and data.

AI Brief

It offers agent builders a benchmark, design principles, and an open-source verifier that improves rubric-based evaluation accuracy while cutting cost.

arXiv:2611.22841v1 Announce Type: new Abstract: As AI agents take on increasingly open-ended and complex tasks, verifying their outputs has proven to be difficult. Rubric-graded agent environments, where a verifier judges multiple natural-language criteria against the deliverables and environment state changes produced by the agent, have emerged as a popular paradigm. We conduct the first systematic study of verifiers for rubric-graded agent environments. First, we create BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks. Next, we develop a methodology that derives from any rubric corpus the capabilities a verifier should possess, yielding a nine-capability taxonomy that we distill into three general design principles---reactive verification, environment alignment, and domain guidance---which we implement in Gandalf, an open-source verifier. Finally, we evaluate Gandalf against three existing verifiers on BVB. Gandalf matches or exceeds every baseline's F1 at lower cost: with Gemini 3 Flash it matches the strongest baseline's F1 (0.864 vs. 0.852) at one-fifth of its cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, where it again achieves the highest F1 (0.955 vs. 0.934 for the best baseline). We release BVB, Gandalf, and the code and data to reproduce our experiments.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!