Live intelligence

Latest AI updates

13 updates

Research Editor selected

MPRBench: A New Benchmark for Multimodal Personalized Reasoning in Vision-Language Models

MPRBench is a comprehensive benchmark introduced to evaluate multimodal personalized reasoning in large language models, covering 12 sub-tasks and supporting interpretable error analysis. Experiments show current MLLMs struggle with recognizing personalized concepts and are sensitive to irrelevant personal information. The benchmark aims to drive further research in this area.

Why it matters: Provides a concrete, recent benchmark with findings that highlight current limitations in MLLM personalized reasoning, valuable for researchers and developers.

arxiv11h agoScore 80

Multimodal Large Language ModelsPersonalized ReasoningBenchmarking

Research Editor selected

NaijaFacts: Benchmark Reveals AI Knowledge Gap on Nigerian Facts

NaijaFacts is a new benchmark of factual and statistical questions about Nigeria, authored by native speakers and available in five languages. Testing eight models shows a dominant knowledge effect: English-query accuracy is only 20% for Nigeria versus 49-50% for US/UK, while language effects are smaller. The gap persists across all 54 African countries, with models scoring 96% on culture and 98% on politics but only 26% on statistics.

Why it matters: Provides concrete evidence of a major AI knowledge gap for African facts, with actionable insights for multilingual and knowledge-aware model development.

arxiv11h agoScore 85

AI evaluationMultilingual NLPKnowledge representation

Research

Task-Efficient Agent Evaluation Using Cached Traces

This paper introduces a method to estimate an agentic system's benchmark performance from a small number of task executions by leveraging rich execution traces. The approach extracts structured summaries with an LLM and embeds them to create representations that preserve task-related signal while suppressing identity-related noise. On SWE-bench Verified and Terminal-Bench 2.0, combining this geometry-based prediction with an item-response model reduces mean absolute error significantly compared to baselines.

arxiv11h agoScore 75

AI evaluationagentic systemsefficient benchmarking

Tutorial Editor selected

Use decision models with Amazon Bedrock AgentCore Evaluations

This post explains how to integrate decision models like Jev and Strands Decider 2B into Amazon Bedrock AgentCore Evaluations via code-based evaluators. It covers the trade-offs between decision models and LLM-as-a-judge, provides a step-by-step validation procedure, and recommends a cascade pattern where decision models screen turns and escalate low-confidence cases to built-in evaluators. The accompanying code sample demonstrates an HR Assistant agent with deterministic, built-in, and decision-model evaluators.

Why it matters: Provides a concrete, recent AWS tutorial with practical guidance on using decision models for agent evaluation, including validation steps and architecture, valuable for developers building AI evaluation pipelines.

AWS Machine Learning Blog22h agoScore 85

AI evaluationagent evaluationdecision models

Research Editor selected

VisSearch-Bench: New Benchmark Evaluates Long-Horizon Visual Search in Multimodal Agents

VisSearch-Bench introduces 204 human-written tasks across 7 domains to evaluate multimodal agents on visual context management during long-horizon search. It covers five task types requiring agents to combine visual details with external information. Benchmarking 16 models shows the best system solves only 26.47% of tasks, highlighting significant room for improvement.

Why it matters: Provides a concrete, recent benchmark with clear results that quantify current limitations in multimodal agent visual search, valuable for researchers and developers.

arxiv1d agoScore 85

multimodal agentsvisual searchbenchmarking

Research

Personas Are Not People: Identity as State, Not Attribute

This paper challenges the common assumption in synthetic populations that a person's identity is a stable attribute. Drawing on personality research and identity economics, the authors prove that fixed-persona simulations cannot reproduce context-dependent population behavior. They propose a person-in-context design and test it on Reddit and clinical-trial recruitment data, finding it produces context-dependent behavior but still falls short of matching real populations.

arxiv1d agoScore 78

AI evaluationsynthetic datapersonality research

Research Editor selected

SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents

SelfImprove-Bench is a new evaluation framework for recursively self-improving (RSI) agents, organizing tasks into three core capabilities—Exploring, Planning, Reasoning—using controllable game environments. It introduces a generalization ladder (G0–G4) to measure how well capability gains transfer beyond training conditions. Experiments show that existing RSI methods achieve inconsistent generalization across these levels, highlighting the need for more robust self-improvement strategies.

Why it matters: Provides a rigorous, standardized benchmark for measuring generalization in self-improving AI systems, a critical gap in current RSI research.

arxiv2d agoScore 88

recursive self-improvementAI evaluationgeneralization

Research Editor selected

LUCA: A Synthetic World for Evaluating Experimental Reasoning in AI Agents

LUCA is a configurable synthetic environment that tests AI agents' experimental reasoning by having them learn unknown regulatory systems through budgeted experimentsholistically. It separates evaluation of experiment selection, prediction, diagnosis, and intervention, revealing that these capabilities dissociate sharply across frontier LLMs. For example, diagnosis varies 7x across models while prediction varies only 1.4x, and agents often rescue systems while misdiagnosing faults. The framework exposes hidden failures like performing the right experiment but failing to use the evidence.

Why it matters: Provides a novel benchmark for dissecting AI agent reasoning processes, with concrete findings on capability dissociation and failure modes, valuable for AI safety and evaluation research.

arxiv2d agoScore 82

AI evaluationexperimental agentsLLM capabilities

Research Editor selected

Four Bridges: Measuring Deception Propensity in a Multi-Agent Game

A new platform and game, Kradle and Four Bridges, measure deception emerging naturally in multi-agent interactions without explicit instructions to deceive. Across eight frontier models, deceptive behavior appeared in 73% of runs, with rates varying from 6% to 100% per model. Interventions like telling all agents that one knows the death room reduced deception in most models, and an honesty steering vector on an open-weight model cut deception to 0%.

Why it matters: Provides a concrete, recent benchmark for evaluating AI honesty in multi-agent settings, with actionable insights for safety interventions.

arxiv2d agoScore 85

AI deceptionmulti-agent systemsAI evaluation

Research

OpenAI Introduces LifeVal Benchmark for Everyday Task Assistance

OpenAI has released LifeVal, a new benchmark designed to evaluate how effectively AI systems assist users with real-world tasks such as finding childcare, managing finances, and handling government paperwork. The benchmark focuses on practical, everyday scenarios rather than abstract reasoning. This initiative aims to measure AI's utility in daily life.

OpenAI News3d agoScore 75

AI benchmarksOpenAItask assistance

Research Editor selected

Pre-Registered Audit of 25 Cognitive-Architecture Modules in an LLM Agent System

The paper introduces a four-question static audit protocol with line-number evidence and call-chain tracing to test whether named cognitive mechanisms in LLM-based agent systems are actually implemented and used. In the audited system claiming 25 cognitive mechanisms, the computed value never reached any production output in 18 modules; the authors identify three idle-wiring patterns: connected-but-never-called, called-but-inert, and called-but-content-blind. Across 61 frozen criteria in 12 interventions, 53 were confirmed, five falsified, two partially adjudicated, and one adjudication-ambiguous. Two repair experiments raised planner task success from 0.028 to 1.000 and cut state-controller error by 52%.

Why it matters: It gives agent developers a reusable audit method and concrete evidence that module names can substantially overstate implemented computation.

arxiv3d agoScore 82

LLM agentscognitive architectureAI evaluation

Research

Audit Finds Test-Time Search Gains Limited for World Action Models

A pre-registered audit of test-time scaling in world action models on Cosmos Policy and RoboCasa reveals that learned value selectors underperform random selection, and oracle gains are minimal. Token pruning reduces FLOPs but fails to achieve target speedups, and a silent rendering corruption was identified and fixed. The study emphasizes auditing test-time scaling as a chain of premises rather than relying on success rates alone.

arxiv3d agoScore 78

test-time scalingworld action modelsAI evaluation

Research

VeriCUA: Verification as Agentic Exploration for Computer-Using Agents

VeriCUA is a platform-agnostic framework that treats task-completion verification for computer-using agents as dynamic evidence gathering and assessment. It first inspects available context to build a task rubric, then selects GUI or code checks for unresolved requirements and combines their findings under explicit evidence rules. The authors report improved agreement with reference labels over compared baselines, with the largest offline gains on hard cases and stronger rejection of false completion claims online.

arxiv6d agoScore 74

computer-using agentsverificationagentic exploration