Back to AI News
arxiv

MM-SystemOneBias Probes Presentation Bias in Multimodal Decision Models

The paper introduces MM-SystemOneBias, a common evaluation protocol that tests 13 models on 5,323 instances drawn from nine public datasets across 11 task scenarios, using six paired interventions: option order, option label, external judgment, image-text description, irrelevant information and rating scale. Across seven matched decision-model–backbone pairs, decision models were more stable under option-label, irrelevant-information and rating-scale changes, and six of seven also showed lower option-order flip rates. They lost less accuracy under incorrect external judgments in all seven pairs and under conflicting descriptions in six, while gaining less from correct external judgments in six pairs. Even where semantic decisions were unchanged, mean absolute shifts in answer probabilities ranged from 4.56 to 11.55 percentage points, exposing input effects that accuracy metrics miss.

AI Brief

It offers researchers and practitioners a reusable protocol and concrete evidence that multimodal decision models can appear stable on accuracy while their output probabilities still shift with irrelevant presentation changes.

arXiv:2611.26557v1 Announce Type: new Abstract: When multimodal decision models determine actions, moderation outcomes, or quality judgments, their reliability depends on preserving semantic decisions across presentation changes, ignoring irrelevant additions, and distinguishing information consistent with task evidence from information that conflicts with it. Existing evaluations study decision-interface biases and multimodal model behavior in separate settings, leaving no common protocol to compare multimodal decision models with their backbones on practical tasks. We introduce \textbf{MM-SystemOneBias}, which evaluates 13 models on 5,323 original instances from nine public datasets and 11 task scenarios using six paired interventions: option order, option label, external judgment, image--text description, irrelevant information, and rating scale. Across seven matched decision-model--backbone pairs, decision models are more stable under option-label, irrelevant-information, and rating-scale changes in all seven pairs; six pairs also show lower option-order flip rates. They incur smaller accuracy losses under incorrect external judgments in all seven pairs and conflicting descriptions in six. In six pairs, they gain less accuracy from correct external judgments and show less favorable accuracy changes under consistent descriptions. Among pairs with unchanged semantic decisions, mean absolute changes in answer probabilities range from 4.56 to 11.55 percentage points, revealing input effects that accuracy alone misses.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!