Back to AI News
arxiv

Study Finds Multimodal Models Struggle to Autonomously Verify Figure-Text Conflicts in Papers

A new arXiv paper evaluates whether multimodal models can autonomously find and diagnose figure-text inconsistencies in full scientific manuscripts. Using 31 manuscripts with hidden reviewer-documented inconsistencies from PRISMM-Bench, six models produced 1,445 reviews; 60.6% mentioned the flawed figure, but only 3.3% were accepted as diagnosing its contradiction by the grading rubric. Naming the figure without stating it was flawed raised rubric-accepted diagnosis from 4.3% to 20.4%, while text-only review produced no accepted diagnosis and a centrality-ranked figure-selection baseline reached 4.8%. The authors argue that deciding what to verify is a major part of the difficulty and that multimodal document understanding should be evaluated on proposing what to verify, not only answering questions about a supplied figure.

AI Brief

It provides concrete evidence that multimodal models are far better at commenting on figures than autonomously identifying which figures contain evidence conflicts, a useful signal for AI evaluation and document-understanding research.

arXiv:2611.24353v1 Announce Type: new Abstract: Scientific figures carry evidence for the claims that cite them, so reviewing a paper means checking that a figure agrees with the text, and not only describing what the figure shows. Benchmarks of figure-text verification typically hand the model the figure, the claim, or the knowledge that an inconsistency exists, which leaves untested whether a model can find the figure worth checking in a complete paper, relate a visual element to the claim it conflicts with, and recognize the conflict. We study this autonomous setting on 31 full manuscripts, each carrying a reviewer-documented figure-text inconsistency from PRISMM-Bench that is hidden from the model. Six multimodal models review each PDF under a neutral prompt, and we then add one input at a time to the same manuscripts. Of 1,445 open-ended reviews, 60.6% mention the flawed figure, yet a grading rubric accepts only 3.3% as diagnosing its contradiction; a blind audit finds the rubric lenient, so we read these as rubric-accepted rates and compare conditions rather than absolute values. In a separate controlled run, naming the figure, without stating that it is flawed, raises rubric-accepted diagnosis from 4.3% to 20.4%, while a consistency instruction and coarse visual cues do not. When the contradiction is visible only in the rendered figure, text-only review yields no accepted diagnosis, while review of the rendered pages with the figure named reaches 15.8%. A baseline that lets the model choose which figures to check, ranked by centrality, stays at 4.8%. Deciding what to verify is thus a large part of the difficulty, and the selection rule we tested does not resolve it. Multimodal document understanding should be evaluated on whether a model can propose what to verify, not only answer questions about a figure it is handed.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!