Live intelligence

Latest AI updates

16 updates

Research Editor selected

MPRBench: A New Benchmark for Multimodal Personalized Reasoning in Vision-Language Models

MPRBench is a comprehensive benchmark introduced to evaluate multimodal personalized reasoning in large language models, covering 12 sub-tasks and supporting interpretable error analysis. Experiments show current MLLMs struggle with recognizing personalized concepts and are sensitive to irrelevant personal information. The benchmark aims to drive further research in this area.

Why it matters: Provides a concrete, recent benchmark with findings that highlight current limitations in MLLM personalized reasoning, valuable for researchers and developers.

arxiv11h agoScore 80

Multimodal Large Language ModelsPersonalized ReasoningBenchmarking

Research Editor selected

A.S.E 2.0: How Repository Context Impacts the Security of LLM-Generated Code

This paper introduces CASE-Bench, a benchmark of 145 repository-level tasks derived from real CVEs across 51 projects, to study how repository context affects the security of LLM-generated code. Experiments with 13 context construction methodshol and six LLMs show that functionality-oriented context improves functional pass rates more than secure pass rates, and that withholding safeguards or inserting insecure examples significantly increases exploitable code. Coding agents observe relevant safeguards in only 27-36% of trajectories, highlighting the need for security-aware context construction.

Why it matters: Provides concrete, recent evidence and a new benchmark on a critical but understudied aspect of LLM code security, offering actionable insights for developers and researchers.

arxiv11h agoScore 85

LLM code securityrepository contextsecure code generation

Research Editor selected

AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document Parsing

AmbiHTMLBench is a new benchmark for evaluating document parsing systems that output HTML instead of Markdown, aiming to preserve fine-grained spatial structure and visual fidelity. It includes 1,115 diverse pages with annotations and uses both objective and subjective evaluation protocols. Tests on ten multimodal systems show that the best objective score is 0.8092, while the subjective winner differs, indicating complementary aspects of parsing quality. Fine-tuning a small expert model shows task-specific adaptation improves document-to-HTML reconstruction.

Why it matters: Introduces a novel benchmark and evaluation methodology for HTML-based document parsing, providing concrete results and insights for researchers and practitioners.

arxiv11h agoScore 80

Document ParsingBenchmarkingMultimodal Systems

Research Editor selected

New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations

CloudOpsBench is a new open-source benchmark that evaluates AI agents on typical DevOps tasks such as fixing services, removing resources, restricting access, rolling out updates, and decommissioning projects. The benchmark identifies reasoning as the primary bottleneck for AI performance in cloud operations, highlighting a key area for improvement.

Why it matters: The article introduces a concrete, recent open-source benchmark with actionable insight into AI limitations in cloud operations, valuable for practitioners and researchers.

The Decoder1d agoScore 82

AI in cloud operationsbenchmarkingreasoning bottleneck

Research Editor selected

EU AI Act Q&A Benchmark Reveals Mixed Readiness of LLM Assistants for Regulatory Tasks

Researchers introduce an expert-curated benchmark of 110 open-ended questions about the EU AI Act to evaluate LLM assistants in regulatory affairs. In the 2026 challenge, only two of five purpose-built assistants outperformed baselines, with one achieving high scores across all metrics. The results indicate that specialist development alone does not guarantee better performance, underscoring the need for rigorous evaluation in regulatory contexts.

Why it matters: Provides concrete benchmark results and insights into LLM reliability for regulatory tasks, valuable for developers and compliance professionals.

arxiv1d agoScore 82

LLM evaluationregulatory complianceAI Act

Research Editor selected

AgenticFinBench: Benchmarking Financial Agents on Diverse Real-World Tasks

AgenticFinBench introduces a benchmark of 230 real-world financial tasks across 8 categories, 2 languages, and 4 complexity levels, with a reference-free evaluation framework that audits hallucinations. Experiments across 18 agent configurations show hallucination auditing lowers the baseline leader's score by 7.58 points, and harness differences explain more variation in evidence grounding than model differences. The benchmark aims to advance financial reasoning and factual verification for reliable agent responses.

Why it matters: Provides a concrete, recent benchmark with novel evaluation methodology and actionable findings for financial AI agents, useful for researchers and practitioners.

arxiv1d agoScore 82

financial AIagent evaluationhallucination auditing

Research Editor selected

GroupAssistBench: Benchmarking Real-World Group Memory Assistance Beyond Recall

GroupAssistBench is a new benchmark for evaluating memory use in group assistance, built from real-world group conversations with a deployed assistant, covering 80,299 messages across 50 groups. It tests both recall (L1) and use (L2) with service requests lacking fixed answers, across single, distributed, and conflicting memories. Evaluations show frontier models and memory systems lag on use, especially for distributed and conflicting memories, but adding retrieval-relation and constraint-binding techniques improves performance.

Why it matters: This introduces a novel benchmark addressing a gap in AI memory evaluation, with real-world data and actionable insights for improving group assistant systems.

arxiv1d agoScore 85

AI memorygroup chat assistantsbenchmarking

Research Editor selected

VisSearch-Bench: New Benchmark Evaluates Long-Horizon Visual Search in Multimodal Agents

VisSearch-Bench introduces 204 human-written tasks across 7 domains to evaluate multimodal agents on visual context management during long-horizon search. It covers five task types requiring agents to combine visual details with external information. Benchmarking 16 models shows the best system solves only 26.47% of tasks, highlighting significant room for improvement.

Why it matters: Provides a concrete, recent benchmark with clear results that quantify current limitations in multimodal agent visual search, valuable for researchers and developers.

arxiv1d agoScore 85

multimodal agentsvisual searchbenchmarking

Research Editor selected

SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents

SelfImprove-Bench is a new evaluation framework for recursively self-improving (RSI) agents, organizing tasks into three core capabilities—Exploring, Planning, Reasoning—using controllable game environments. It introduces a generalization ladder (G0–G4) to measure how well capability gains transfer beyond training conditions. Experiments show that existing RSI methods achieve inconsistent generalization across these levels, highlighting the need for more robust self-improvement strategies.

Why it matters: Provides a rigorous, standardized benchmark for measuring generalization in self-improving AI systems, a critical gap in current RSI research.

arxiv2d agoScore 88

recursive self-improvementAI evaluationgeneralization

Research Editor selected

KAN vs. MLP Benchmark on Clinical Tabular Data Shows Trade-offs in Precision vs. Efficiency

A rigorous benchmark compares nine KAN variants against MLPs and Random Forest across seven medical datasets. Results show KANs (especially TabKAN and ReLUKAN) improve minority-class detection on imbalanced data but require 1.2x–1.48x longer inference and up to 12.5x more parameters. StandardMLP remains the efficiency Pareto frontier, while KANs are viable when precision on rare positive cases is the priority.

Why it matters: Provides the first large-scale, statistically rigorous comparison of KANs vs. MLPs on clinical tabular data, offering actionable guidance for practitioners.

arxiv2d agoScore 88

Kolmogorov-Arnold NetworksMulti-Layer Perceptronsclinical tabular data

Research Editor selected

Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models

This paper introduces a hierarchy of word problems for state tracking, from simple immutable assignments to general finite-state tracking, and proves that transformers cannot length-generalize on all aperiodic problems while Gated DeltaNet can. The authors build StateBench, a 4-tier benchmark, and show that both trained and pretrained models, including frontier ones, fail to robustly solve even soluble state tracking.

Why it matters: Provides a new theoretical hierarchy and benchmark that clarifies architectural differences in state tracking, with practical implications for model evaluation and development.

arxiv2d agoScore 85

state trackinglanguage model architectureslength generalization

Research Editor selected

ARCBench: Benchmarking Rule Compliance in LLM Agents Under Context Compaction

ARCBench is a new benchmark built on the pi agent harness to measure rule compliance in tool-calling agents after context compaction. It reveals that most violations occur while the rule is still present in the compacted history, often because agents treat later requests as permission. The benchmark also shows that compaction strategy and rule delivery affect violation sources, and that retaining a rule does not guarantee compliance.

Why it matters: Provides a concrete, recent benchmark with actionable findings on a critical failure mode in deployed LLM agents, useful for researchers and practitioners.

arxiv2d agoScore 82

LLM agent reliabilitycontext window managementrule following

Research Editor selected

ScienceCUA: Turning Documentation into Evolving Skills for Scientific Computer Use Agents

ScienceCUA is a two-stage framework that converts software documentation into executable, reusable skills for scientific computer-use agents. It first constructs skills from manuals and web pages, then uses execution feedback to evolve those skills by contrasting successful and failed trajectories. The framework also introduces two improved benchmarks, ScienceBoard-Verified and ScienceBoard-VE, to reduce noise and leakage. Experiments show up to 17.12 percentage points improvement in task success rate over the base model.

Why it matters: This paper presents a concrete method and benchmarks that improve AI agents' ability to use scientific software, with measurable gains and practical implications.

arxiv2d agoScore 82

AI agentsscientific computingskill acquisition

Research Editor selected

DeliverBench: Benchmarking AI Agents on Multi-Stage Professional Project Delivery

DeliverBench is a new benchmark that evaluates AI agents on completing realistic multi-stage professional projects, such as financial due diligence, legal contract drafting, and drug safety reporting. It includes 11 projects reconstructed with domain experts, chaining 150 subtasks across multiple rounds of feedback. Results show that even the best system scores only 65.0/100 on final deliveryaine, highlighting that task-level capability does not simply compose into full project delivery.

Why it matters: This introduces a novel benchmark addressing a critical gap in AI agent evaluation, with concrete results showing limitations in current systems.

arxiv2d agoScore 85

AI agent evaluationprofessional project deliveryLLM capabilities

Research Editor selected

Grid Agents: An Agentic Framework for Automated Power System Studies and Grid Connection Assessment

The paper introduces Grid Agents, an AI system combining a knowledge ontology (Grid Ontology) and an engineering agent (Grid Studio) to automate power system studies and grid connection assessments. It includes a benchmark suite of 519 tasks from eight power-system familiesched, with verification via replaying tool calls. The system outperforms general coding agents and retrieval pipelines on regulatory evidence tasks and leads on several RAG leaderboards, while Grid Studio achieves top scores and faster performance on power-system tasks.

Why it matters: This is a concrete, recent AI development with a novel agentic framework, benchmark suite, and measurable results, offering practical value for power system engineering and AI evaluation.

arxiv2d agoScore 85

AI agentspower system engineeringretrieval-augmented generation

Research

Implicit Assumptions About Tasks in Continual Learning

This paper clarifies that continual learning measurements depend on factors like data similarity, change rate, and order, which can be defined at different levels: underlying data, partition, or learner. The authors show these levels often don't coincide and are usually left implicit, leading to ambiguous results. Through teacher-student experiments, they demonstrate that changing granularity, order, or similarity structure alters measured outcomes. They also review benchmarks to see which properties are stated, noting that for foundation models with undisclosed data, properties are inherited by the model's initial state.

arxiv3d agoScore 72

continual learningevaluation methodologytask design