BlackSwan: Refutation-Based Action Verification for Language Model Agents
BlackSwan is a new verification procedure for language model agents that checks proposed actions by actively seeking evidence that refutes them, rather than only confirming supporting conditions. It predicts intended effects, formulates testable hypotheses for failure, and tests them before execution. On AppWorld, it improves success rates by up to 25.2 percentage points, and on WebShop by 12.7 points, even without thinking mode or demonstrations.
Why it matters: This introduces a novel, effective verification paradigm with concrete benchmark gains, offering practical value for AI agent safety.