LLM-Generated World Models Can Be Mechanically Valid but Structurally Wrong
A new arXiv study evaluates whether five LLMs can generate formal world models for neuro-symbolic reasoning as partially observable stochastic games. It introduces a 44-scenario benchmark and analyzes 410 first-pass generations across mechanical validity, reference-relative structural agreement, and semantic criteria. In the game track, 134 of 135 models passed a strict mechanical audit, but mean structural F1 was only 0.366 and just 28 of 134 exactly matched the authored reference. Naming a country in the cultural track changed structural representations for most LLMs while producing only small semantic-score changes.
It shows that executable LLM-generated world models may still diverge structurally from intended decision problems, a key limitation for neuro-symbolic planning and verification.
Comments & Questions (0)
No comments yet
Be the first to comment!