MCPHallu: New Benchmark Exposes Hallucination-Prone Failures in MCP Agents
MCPHallu is a new benchmark designed to diagnose hallucination-prone failures in LLM agents using the Model Context Protocol (MCP). It includes 358 tasks across five domains, each with structural triggers for four failure subtypes: Branch Collapse, Unreachable Goal, Tool Misuse, and Context Forgetting. Evaluation of 14 frontier agents from eight vendors on 5,012 trajectories shows an average score of 0.642, with particular weakness on Unreachable Goal tasks (0.458 average, 23.2% pass rate). The benchmark enables subtype-level analysis, revealing that model rankings shift significantly across subtypes.
Why it matters: Provides a new diagnostic benchmark that goes beyond task completion to identify specific hallucination failure modes in MCP agents, offering actionable insights for improving agent reliability.