Back to AI News
arxiv

Pre-Registered Audit of 25 Cognitive-Architecture Modules in an LLM Agent System

The paper introduces a four-question static audit protocol with line-number evidence and call-chain tracing to test whether named cognitive mechanisms in LLM-based agent systems are actually implemented and used. In the audited system claiming 25 cognitive mechanisms, the computed value never reached any production output in 18 modules; the authors identify three idle-wiring patterns: connected-but-never-called, called-but-inert, and called-but-content-blind. Across 61 frozen criteria in 12 interventions, 53 were confirmed, five falsified, two partially adjudicated, and one adjudication-ambiguous. Two repair experiments raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%.

AI Brief

It gives agent developers a reusable audit method and concrete evidence that module names can substantially overstate implemented computation.

arXiv:2610.41604v1 Announce Type: new Abstract: Agent systems built on local large language models are increasingly marketed through the vocabulary of cognitive architecture: modules named global workspace, theory of mind, causal reasoning, or active inference are listed as features, and the presence of the name is taken as the presence of the mechanism. We ask the prior question -- does the named computation actually happen -- and contribute a transferable verification method for the claim-implementation gap: a four-question static audit protocol with line-number evidence and call-chain tracing, demonstrated on two systems. We audit a system claiming 25 cognitive mechanisms, pre-register falsifiable predictions for 11 modules and execute the interventions, replicate the protocol on the IIT-phi core of a second, independent system with its own pre-committed falsification criteria, and run two repair experiments as positive controls. In 18 of 25 modules the computed value never reaches any production output; three idle-wiring shapes new to the dead-code and code-smell literatures emerge (connected-but-never-called, called-but-inert, called-but-content-blind). Of 61 frozen criteria across 12 interventions, 53 were confirmed, two partially adjudicated, five falsified, one adjudication-ambiguous. Repairs raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%. Module names and module computations must be audited as independent claims; the protocol and the five-class failure taxonomy are reusable for the claim-implementation gap at two-system, case-level depth.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!