Back to AI News
arxiv

MuSeToM benchmark tests LLM belief revision across six sensory cues

Researchers introduce MuSeToM, a human-validated text-based benchmark for evaluating epistemic belief revision in Theory of Mind reasoning. It contains 2,415 instances adapted from BigToM and OpenToM, with textual cues spanning six sensory categories. The benchmark tests whether models can decide when a protagonist should revise a belief and infer the updated belief state. Evaluations under direct and chain-of-thought prompting show models often detect possible belief change but struggle to identify the updated belief and over-update on salient but insufficient cues.

AI Brief

It provides a validated benchmark and identifies a specific failure mode in LLM social reasoning, useful for researchers evaluating Theory of Mind capabilities.

arXiv:2611.16662v1 Announce Type: cross Abstract: Theory of Mind (ToM), the ability to infer mental states in others, is pivotal for human social cognition. Existing evaluations of ToM in large language models (LLMs) mainly focus on static belief attribution and false-belief reasoning, leaving a critical aspect of social reasoning underexplored: can LLMs decide whether perceptual evidence is sufficient to justify another person's belief revision? To address this gap, we present MuSeToM, a human-validated, text-based benchmark for evaluating epistemic belief revision in ToM reasoning. MuSeToM contains 2,415 instances adapted from BigToM and OpenToM, with textual cues spanning six sensory categories, and tests whether models can determine when a protagonist should revise a belief and infer the resulting updated belief state. Using MuSeToM, we systematically evaluate state-of-the-art LLMs under direct and chain-of-thought prompting, revealing a pronounced dissonance: while models often detect that a belief may change, they struggle to identify the updated belief and frequently over-update on perceptually salient but epistemically insufficient cues.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!