ReA-Bench: A Diagnostic Benchmark for Auditing Responsible Agent Behavior
The authors argue that aggregate recall hides which risks agent auditors miss and whether higher recall comes with more false positives on legitimate behavior, and that existing benchmarks offer limited insight into these failures. They introduce ReA-Bench, which organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. The benchmark includes challenging benign cases whose actions resemble risky ones but are consistent with the task and its authorization, plus paired trajectory versions that test whether shortening a trajectory or supplying missing information makes risk detection more reliable. Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks.
It gives agent-platform and safety teams a diagnostic benchmark that separates missed risks from false positives, which overall recall alone cannot reveal.
Comments & Questions (0)
No comments yet
Be the first to comment!