Back to AI News
arxiv

Audit Finds Test-Time Search Gains Limited for World Action Models

A pre-registered audit of test-time scaling in world action models on Cosmos Policy and RoboCasa reveals that learned value selectors underperform random selection, and oracle gains are minimal. Token pruning reduces FLOPs but fails to achieve target speedups, and a silent rendering corruption was identified and fixed. The study emphasizes auditing test-time scaling as a chain of premises rather than relying on success rates alone.

AI Brief

Provides concrete, recent empirical evidence on the limitations of test-time scaling for AI control models, valuable for researchers and practitioners.

arXiv:2610.41970v1 Announce Type: new Abstract: World-action models increasingly apply test-time scaling, spending extra inference computation on candidate selection and adaptive allocation to improve control. Such gains are credible only when candidates carry exploitable signal, nominal compute savings become real speedup, and evaluation stays reliable. We audit all three on Cosmos Policy and RoboCasa with a premise-first, pre-registered methodology. On clean data, a learned value selector is 1.66 pp worse than random; a hindsight oracle gains only 1.40 pp, with an episode-level interval spanning zero; and 82% of states show identical candidate outcomes, leaving little selection headroom. Token pruning cuts per-forward FLOPs by up to 47% but yields at most a measured 1.15$\times$ end-to-end speedup on the deployed path (1.25$\times$ on a decode-excluded path), short of a 1.5$\times$ target, and our adaptive-allocation experiment is inconclusive under a power gate. We further identify and fix a silent rendering corruption whose rate rises sharply along compute-heavy planning paths while leaving the official direct-policy result unaffected, and release a detector, fix, and scripts. These findings show why test-time scaling should be audited as a chain of premises (decision signal, realized efficiency, evaluationintegrity) rather than by success rate alone. Conclusions are scoped to the tested configuration.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!