AGI-Eval

AGI-Eval is a research-oriented, community-governed evaluation ecosystem designed to advance the scientific, fAIr, and comprehensive assessment of artificial general intelligence (AGI)-aligned capabilities in foundation models. Launched through a coalition of leading Chinese academic institutions—including Shanghai Jiao Tong University, Tongji university, East China NORMal UniveRSIty, and the DataWhale open-source commUnity—it operates as a neutral, open infrastructure for benchmarking model intelligence across cognitive, linguistic, and problem-solving dimensions.

What it does

AGI-Eval does not generate content or serve as an end-user APPlication. Instead, it functions as a rigorous eValuation layer for AI models—especially large language models—by DeFining Standardized tasks that mirror human reasoning, decision-making, and domain knowledge. Its mission centers on measuring how well models perform on exams and challenges that reflect real-world cognitive demands, thereby bridging the gap between technical performance metrics and prACTical utility.

Key capabilities

• Public Leaderboard: Publishes regularly updated rAnkings of major LLMs across composite scores and granular ability categories—such as logical reasoning, mathematical Fluency, multilingual comprehension, and commonsense inference—with full transparency into scoring methodology and dataset provenance.
• Evaluation Suite Management: Hosts three tiers of test collections: open academic benchmarks (e.g., MMLU, C-Eval), proprietary AGI-Eval–designed assessments spanning diverse domains, and user-subMITted custom evaluations—enabling researchers to extend and specialize the evaluation landscape.
• Data Studio Infrastructure: Supports large-scale, high-quality data curation via a 30,000+ contributor crowdsourcing network, Multi-modal annotation Workflows (single-item, expansion, arena-style pairwise Comparisons), and dual-layer quality control combining automated validation and expert human review.
• Human-AI Co-Evaluation Framework: Facilitates structured competitions where humans and models jointly tackle tasks, generating rich behavioral insights beyond scalar scores—such as error patterns, calibration gaps, and contextual Adaptability.

Who it is for

AGI-Eval serves AI researchers, model developers, academic labs, and open-source contributors focused on objective, reproducible model assessment. It is especially valuable for teams building or fine-tuning LLMs who need reliable baselines, comparative analysis tools, and community-validated benchmarks—not for non-technical users seeking creative or Productivity Tools.

How to get started

Access is web-based and free. Users can browse live leaderboards, download public evaluation datasets, submit new test cases, or join the crowdsourcing pool via the official portal. Integration with external models typically requires API or local Inference setup; documentation and contribution guidelines are available in Chinese, with some English resources emerging gradually. No registration is mandatory for viewing, but participation in evaluations or submissions may require account creation.

Pricing and access caveat

AGI-Eval offers unrestricted free access to its core services, including leaderboard data, open datasets, and evaluation frameworks. While no paid tiers are publicly documented, certain advanced features—such as private dataset hosting, priority review queues, or Enterprise-grade analytics—may vary by partnership or institutional agreement. As a research initiative, its long-term sustainability model remains community- and grant-supported; commercial usage terms may evolve. All hosted data and tools are intended for non-commercial, academic, and open-science purposes unless otherwise specified.