Live intelligence

Latest AI updates

1 updates
Research

Gandalf: An Open-Source Verifier for Rubric-Graded Agent Environments

This arXiv paper presents the first systematic study of verifiers used in rubric-graded agent environments, where a verifier judges natural-language criteria against agent deliverables and environment state changes. The authors introduce BankerVerifierBench (BVB), a meta-evaluation dataset of 2,464 human-judged rubric criteria across 24 investment-banking tasks, and derive a nine-capability taxonomy of verifier requirements distilled into three design principles: reactive verification, environment alignment, and domain guidance. Implemented as the open-source verifier Gandalf, it matches or exceeds three existing verifiers on BVB at lower cost — with Gemini 3 Flash it reaches F1 0.864 versus 0.852 for the strongest baseline at one-fifth the cost, and with GPT-5.4 models it improves F1 by over 0.12 at lower cost. Gandalf also generalizes to OpenClaw, a structurally different personal-productivity environment, achieving the highest F1 there (0.955 vs. 0.934 for the best baseline), and the authors release BVB, Gandalf, and reproduction code and data.

AI agent evaluationrubric-graded environmentsverifier models