Rel-Bloom: Multilingual Benchmark Reveals LLM Reasoning Gaps over Supplied Relational Facts
This paper introduces Rel-Bloom, a controlled multilingual diagnostic of 450 multiple-choice questions built from 30 Wikidata relations, testing evidence-conditioned relational reasoning across five Bloom-inspired levels and three veracity conditions. Across 43 models, six languages, and 226,800 predictions, it finds sharp accuracy drops from Apply to Analyze in open-weight models, persistent bottlenecks on Evaluate items even for strong models, and cross-language consistency in difficulty rankings. The results show that greater model capability does not guarantee reliable reasoning over supplied relational evidence.
Why it matters: Provides concrete, recent benchmark results and insights into LLM reasoning limitations over non-parametric memory, valuable for model evaluation and development.