Back to AI News
arxiv

HumanSightBench: A Benchmark for Human Visual Perception in MLLMs

HumanSightBench is a new benchmark of 2,876 short-answer visual questions spanning 7 top-level perceptual categories and 21 subcategories. Items pass five inclusion gates—question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering—to reduce ambiguity and external-knowledge confounds. Across eight evaluated MLLMs, the best pass@3 score was 65.6%, and no model exceeded 27.5% on Counting. The results expose perception gaps that overall benchmark scores can obscure.

AI Brief

It provides a carefully filtered benchmark that isolates human-easy visual perception failures in MLLMs, revealing capability gaps hidden by aggregate scores.

arXiv:2611.12305v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have demonstrated strong performance on knowledge-intensive and reasoning-based vision-language benchmark datasets, but still struggle to solve the visual questions that people can solve easily. Evaluating the differences between MLLMs and human-level visual perception is challenging due to end-to-end evaluation which combines various skills, or because we are uncertain if individual items are valid. Specifically, observed errors may reflect visual perception, language priors, background knowledge, question ambiguity, and underspecified answers. Our guideline for HumanSightBench is that we construct items that are human-easy by construction and model-hard by evaluation. HumanSightBench contains 2,876 visual questions with short answers across 7 top-level perceptual categories and 21 subcategories. We only include items that pass five inclusion gates: question clarity, answer uniqueness, visual grounding, human solvability, and text-only shortcut filtering. According to the protocol, errors are defined as incorrect answers on human-validated visual tasks with the evaluated interface. The inclusion gates are intended to reduce confounds from ambiguity and external knowledge. The best pass@3 score is 65.6% across eight evaluated MLLMs. No evaluated model exceeds 27.5% on Counting. These results identify capability gaps that are hidden by overall scores and can help future training experiments and held-out evaluation.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!