ConceptARC at Scale: New Benchmark and Analysis of Conceptual Understanding in Reasoning Models
The paper introduces ConceptARC at Scale, a generator that creates thousands of ARC-style tasks targeting specific spatial and semantic concepts, and presents ConceptARC-100, a dataset of 1,600 manually reviewed tasks across 16 concepts—ten times the size of the original benchmark. Evaluations of recent large reasoning models and models trained from scratch show which concepts are solved reliably and where they struggle. Internal representation analysis reveals that task representations cluster by transformation operations (e.g., recoloring, removal, translation), suggesting models form stable abstractions that generalize across diverse tasks.
Why it matters: Provides a new large-scale benchmark and insights into how reasoning models abstract concepts, valuable for AI researchers and practitioners.