Live intelligence

Latest AI updates

17 updates
Research

The Between-Round Decision in Autonomous Research: An LLM Controller That Reads the Whole Trajectory

The paper introduces BRED (Between-Round Evidence-based Decision), an LLM-based controller for autonomous research that decides after each experiment round whether to refine the current best solution or start a new direction, which solution to build from, and what goal to set. It maintains a Champion solution and a log of past rounds, making one decision per round without a search structure. On ETTh1, SST-2, and CIFAR-100, BRED reports improvements over task baselines, and ablations show that reading accumulated evidence improves results. It also reports a macro-F1 of 0.6164 on speech-based depression detection under a leakage-free protocol.

autonomous research agentsLLM controllersbetween-round decision
Research

OmniScore-Bench Evaluates Vision-Language Models on Musical Score Reading

A new arXiv paper introduces OmniScore-Bench, a benchmark for assessing vision-language models on optical music recognition and score understanding. It includes 3,000 questions across six aspects and 24 task subtypes, covering symbol localization, transcription, musical consistency, and evidence-linked question answering. The authors evaluate twelve VLMs and analyze how image access, auxiliary context, and page scope affect performance.

Vision-language modelsOptical music recognitionBenchmark evaluation
Research

PolyRoute Benchmark Tests Typed-Decision Routing Across 12 Languages and Code-Mixed Text

PolyRoute is a new benchmark of 322 native-language support messages in 12 languages and 9 writing systems, plus 156 code-mixed or romanised messages such as Hinglish, Spanglish and Arabizi, for a three-way department-routing decision. Labels were produced by Qwen3.8-27B with a blind second pass, and an independent model agreed on 97.2% of test items. A leaderboard of 13 systems reports 61.2% for the English-root checkpoint, 77.3% for the shipped multilingual checkpoint and 96.6% for the best fine-tune, while TF-IDF with logistic regression reaches 91.9%. The authors note that the split resolves differences only above about 4 points and that all texts and labels are LLM-generated.

PolyRoutemultilingual NLPbenchmark evaluation
Research

An Augmented Lagrangian Approach to Lifted Bellman Linear Programming

This paper introduces AL-LBLP, an augmented Lagrangian method for solving Lifted Bellman Linear Programming in offline reinforcement learning. Instead of fixed hinge-penalty coefficients, it learns multipliers via dual ascent with a single penalty parameter and shows that in the convex case fixed points are KKT pairs of LBLP. The critic can drive a Gaussian actor, a flow actor, or best-of-N sampling, and for the Gaussian actor the authors propose an advantage-normalized DDPG+BC objective. Using one Q and V and no target network, AL-LBLP with best-of-N reports 60.1% average success across 50 state-based OGBench tasks, 81.5% on humanoidmaze-giant, 79.8% on antmaze-giant, and no collapse up to batch size 8192.

offline RLoptimizationbenchmark evaluation
Research

Study compares weight quantization against option-order averaging under a fixed inference budget

The paper asks whether lower-bit weight storage degrades the benefit of averaging answer-option reorderings, and which use of a fixed budget is cheaper. Using simulated round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B, evaluated on ARC-Easy, ARC-Challenge, MMLU and HellaSwag with 300 items each, the authors report that 6-bit weights lost no accuracy against fp16 while 4-bit raised answer-slot skew. Averaging four cyclic orders recovered +12.6 points at 4 bits for the 0.6B model. In cost-matched pairs measured as weight bytes read times passes, extra orders beat extra bits only at 6 or more bits, leading to a suggested rule of 6-bit weights plus up to four order passes.

quantizationLLM inference efficiencysmall language models
Research

Zero-Search Reinforcement Learning for Slay the Spire 2 with Failure-Driven Restarts

Researchers trained a 6.6M-parameter entity transformer to play Slay the Spire 2 with one forward pass per action and no lookahead, using the game's compiled assembly in a headless .NET host. A failure-driven curriculum restarts training episodes from visited states in proportion to where complete runs are lost. On 12,800 held-out Ascension 10 games, the released policies averaged 8.05% wins across five characters, while AlphaSpire's networks without search won 0.55% on a common panel. At Ascension 0 the policies won 80.2%, close to Spire Mentor's reported 80.3%, though Mentor's search led at higher ascensions and by 8.8% to 8.2% on 2,000 A10 games in the same host.

Slay the Spire 2reinforcement learninggame AI
Research

Joi-Bench-Task benchmarks task coordination in full-duplex voice agents

The paper introduces Joi-Bench-Task, a benchmark designed to isolate coordination between task execution and conversation in full-duplex voice agents. It uses an interactive user simulator that injects configurable interventions during tool execution and spoken result delivery for reproducible evaluation. Five commercial voice agents are assessed on execution responsiveness, coordination under request changes, and coordination under interruptions. The reported results show a maximum 55% result completion rate under request changes and average information omission rates of at least 23% after interruption for four of the five agents.

full-duplex voice agentsbenchmark evaluationhuman-agent interaction
Research

ABHBench: Evaluating Moral Decision-Making of Foundation Models from an Agentic Perspective

Researchers introduce ABHBench, a multimodal benchmark built from the interactive narrative game Detroit: Become Human to evaluate foundation models as first-person moral decision-makers. The benchmark places models in branching scenarios with visual observations, dialogue, memory, and action choices, measuring Harmlessness, Obedience, Self-preservation, Autonomy, Stability, and Certainty. Across 12 foundation models, 75% shifted significantly between a text-only abstraction and the matched multimodal scenario. Reasoning-enhanced models showed no monotonic gains in consistency or human-oriented behavior.

AI safetyagentic AIbenchmark evaluation
Research

When Feedback Becomes a Backdoor: Tracing Backdoor Skill Formation in Text-Space Skill Optimization

This arXiv paper studies reference-feedback poisoning in SkillOpt, where poisoned reference answers can induce trigger-conditioned rules in reusable text Skills without attacker access to the model, optimizer, or Skill file. In SearchQA, the natural attack retained an exact rule across all three seeds, with final Skills emitting the payload on 200/200 triggered questions and 0/200 clean questions; removing the rule in seed 42 eliminated activation, and clean accuracy was similar to benign SkillOpt (169/200 versus 171/200). In SpreadsheetBench, no natural run retained an exact rule across three seeds, with diagnostics identifying literal abstraction by the Analyst and edit-budget ranking as successive bottlenecks. The findings show that poisoned references can create persistent backdoors, but exposure alone does not guarantee that an induced rule is formed, selected, and retained.

AI agent securitybackdoor poisoningtext-space skill optimization
Research Editor selected

LLM Repairs Robot Policy by Editing Token-Embedding Weights

The paper applies weight-space programming to a robot foundation policy: a language model reads the policy's weights and edits token-embedding rows using a CPU workspace and budgeted simulator feedback, with no gradient updates or new demonstrations. Across three independent sessions on π0.5, success on 765 held-out paraphrases improved, with the best run rising from 86.3% to 92.0% (48 repairs, four regressions, exact paired p=1.31e-10). Gains concentrate on the 335 instructions containing edited tokens (74.3% to 87.5%), while matched random-direction edits do not help. The authors characterize the result as precise lexical repair rather than general acquisition of missing robot capabilities, and separately report that benchmark metadata lowered FastWAM success from 70.0% to 48.6%.

Why it matters: It offers a concrete, unusually detailed demonstration that an LLM can fix a robot policy's instruction grounding by editing a handful of embedding rows, plus a cautionary evaluation finding about benchmark metadata effects.

roboticsmodel editingLLM agents
Research

Study: Exposing Success Criteria to Embodied Agents May Cause Overfitting, Not Better Task Learning

A new arXiv paper investigates the practice of exposing benchmark success criteria to self-improving embodied agents, a method called Criteria Exposure (CE). Controlled experiments show that agents with CE achieve higher scores under the given criteria, but the advantage disappears when criteria are slightly altered without changing instructions. The skills developed under CE transfer poorly across tasks, suggesting overfitting to the verifier rather than learning the task. The authors propose instruction-faithful criteria to better reflect actual task performance.

embodied agentsbenchmark evaluationoverfitting
Research Editor selected

FrontierPhysics: A New Benchmark for End-to-End AI Physics Research

FrontierPhysics is a benchmark created by 42 physicists with 56 tasks spanning theoretical, computational, and experimental physics. It evaluates AI agents on the full research pipeline—literature review, planning, experimentation, and manuscript writing—using both deterministic verifiers and expert rubrics. Across 22 models in 26 configurations, no agent passed more than 46 of 168 rollouts, with the main weaknesses being insufficiently rigorous reasoning and limited expert intuition rather than knowledge gaps.

Why it matters: Provides a rigorous, expert-curated evaluation of frontier AI agents' ability to conduct end-to-end physics research, highlighting critical reasoning deficiencies.

AI agentsphysics researchbenchmark evaluation
Research Editor selected

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

BEAKER is a new benchmark with 1,000 expert-curated QA samples across 11 task types to evaluate multimodal LLMs as embodied brains in self-driving chemical labs. It tests affordance grounding, trajectory planning, equipment/safety recognition, and procedural reasoning. Evaluations of 28 MLLMs reveal that while models grasp coarse procedures, they struggle with fine-grained action grounding and precise perception. The benchmark highlights a critical gap for future embodied AI in laboratory automation.

Why it matters: Provides a rigorous, expert-curated diagnostic benchmark that exposes specific weaknesses in current MLLMs for embodied chemical lab tasks, guiding future research.

embodied AImultimodal large language modelsself-driving chemical laboratories
Research Editor selected

CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling

CAREFUL-Bench enhances the Berkeley Function Calling Leaderboard (BFCL) by making evaluation effect-aware without replacing it. It infers effect declarations for all 128 functions via simulator execution and uses a judge-free replay scorer that reproduces official results while reporting unrequested destructive calls, duplicate writes, redundant reads, and token costs. Across 11 open models, effect-aware scoring preserves ranking but reveals quality gaps, and a confirmation-aware protocol shows models rarely obtain explicit permission before destructive actions.

Why it matters: Provides a concrete, recent improvement to function-calling benchmarks, exposing critical limitations in current evaluation and offering actionable metrics for model developers.

LLM function callingbenchmark evaluationagent safety
Research Editor selected

Auto-Research Agent Optimizes MSA for Protein Complex Prediction

A new autonomous agent treats AlphaFold 3 as a black-box instrument to adaptively reorganize MSAs for protein-protein complex prediction. It improves interface DockQ across three benchmarks while using 60% fewer MSA sequences on average, with largest gains for inter-species heteromers. The method shows MSA design is target-specific and can be optimized via self-directed experimentation.

Why it matters: Provides a concrete, recent AI-driven method with benchmarked improvements in a challenging bioinformatics task, offering practical value for researchers.

protein structure predictionmultiple sequence alignmentautonomous experimentation
Research Editor selected

M4Bench Evaluates AI Coding Agents on Clinical EHR Derivations

Researchers introduced M4Bench, a benchmark of 28 clinical derivation tasks across 15 task families using MIMIC-IV and eICU, to test whether AI coding agents can reproduce EHR-derived variables and how clinician-reviewed agent skills affect results. In a Docker-isolated Codex campaign, skills raised reward by 0.136 over the no-skill baseline, with 26 of 28 tasks improving. Specification completeness accounted for most of the gain: an operational spec alone raised reward by 0.162 on an eight-task subset, while skills added only 0.024. The authors note the gains reflect partial agreement with reference derivations rather than completed tasks, and release tasks, skills, evaluation code, and sanitized per-run evidence as an open-source benchmark.

Why it matters: It provides a concrete benchmark and evidence on where coding agents help or fail in clinical EHR derivation, showing that precise specifications matter more than generic agent skills.

M4Benchclinical EHR derivationAI coding agents
Research

Google Open-Sources Harness², a Recursive Agent Harnessing Framework

Google Cloud AI Research has released Harness², an open-source framework (Apache-2.0, v0.1.0) that refines an agent's harness—prompts, tools, memory, skills, orchestration and other components—at deployment time while keeping model weights frozen. It uses task-level contrastive "Propose, Probe, Compose" recursion plus a cross-task lessons library to improve agents without ground-truth labels. In the paper's benchmarks, a single refinement step improved all four model/executor setups on three pro

Google Cloud AI ResearchAI agentsagent harness optimization