Live intelligence

Latest AI updates

1 updates
Research

Study Finds Multimodal Models Struggle to Autonomously Verify Figure-Text Conflicts in Papers

A new arXiv paper evaluates whether multimodal models can autonomously find and diagnose figure-text inconsistencies in full scientific manuscripts. Using 31 manuscripts with hidden reviewer-documented inconsistencies from PRISMM-Bench, six models produced 1,445 reviews; 60.6% mentioned the flawed figure, but only 3.3% were accepted as diagnosing its contradiction by the grading rubric. Naming the figure without stating it was flawed raised rubric-accepted diagnosis from 4.3% to 20.4%, while text-only review produced no accepted diagnosis and a centrality-ranked figure-selection baseline reached 4.8%. The authors argue that deciding what to verify is a major part of the difficulty and that multimodal document understanding should be evaluated on proposing what to verify, not only answering questions about a supplied figure.

multimodal document understandingscientific figure verificationAI evaluation benchmarks