MM-SystemOneBias Probes Presentation Bias in Multimodal Decision Models
The paper introduces MM-SystemOneBias, a common evaluation protocol that tests 13 models on 5,323 instances drawn from nine public datasets across 11 task scenarios, using six paired interventions: option order, option label, external judgment, image-text description, irrelevant information and rating scale. Across seven matched decision-model–backbone pairs, decision models were more stable under option-label, irrelevant-information and rating-scale changes, and six of seven also showed lower option-order flip rates. They lost less accuracy under incorrect external judgments in all seven pairs and under conflicting descriptions in six, while gaining less from correct external judgments in six pairs. Even where semantic decisions were unchanged, mean absolute shifts in answer probabilities ranged from 4.56 to 11.55 percentage points, exposing input effects that accuracy metrics miss.