Same Outputs, Different Scores: Auditing the Unchecked Graders of Multimodal LLM Benchmarks
This paper audits the scoring rules used in multimodal LLM benchmarks, showing that the choice of grader can swing accuracy by up to 22 points on identical outputs. The authors reproduce official scorers of eight benchma...