Pre-Registered Audit of 25 Cognitive-Architecture Modules in an LLM Agent System
The paper introduces a four-question static audit protocol with line-number evidence and call-chain tracing to test whether named cognitive mechanisms in LLM-based agent systems are actually implemented and used. In the audited system claiming 25 cognitive mechanisms, the computed value never reached any production output in 18 modules; the authors identify three idle-wiring patterns: connected-but-never-called, called-but-inert, and called-but-content-blind. Across 61 frozen criteria in 12 interventions, 53 were confirmed, five falsified, two partially adjudicated, and one adjudication-ambiguous. Two repair experiments raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%.
It gives agent developers a reusable audit method and concrete evidence that module names can substantially overstate implemented computation.
Comments & Questions (0)
No comments yet
Be the first to comment!