Live intelligence

Latest AI updates

5 updates
Research Editor selected

Benchmark Shows Stored Preferences Can Make AI Assistants Silently Omit Decision-Critical Information

Researchers introduce an Inspect benchmark of 100 everyday decisions across 25 domains in which only the assistant's memory of the user varies. Testing five models, they find that a preference for very short answers and no warnings can be stored from a single line, leading later answers to silently miss critical items. The same models then select options for users on most value-dependent choices when such memory is present, versus none with neutral memory. Prompting models to list must-know points before applying the user profile restores more content in four of five models.

Why it matters: It provides a concrete benchmark and evidence that assistant memory and preference personalization can silently degrade decision-critical answers, which matters for teams building or evaluating AI assistants.

AI assistantsmemorybenchmark
Research

OICycle: Observe, Infer, and Intervene Cycle for Belief-State Learning in Longitudinal Clinical Decision Making

OICycle is a method for learning a stochastic belief state directly from retrospective clinical trajectories, treating clinical care as a partially observable Markov decision process. At each episode, a learned recurrent operator fuses a sample of the previous belief with new multimodal observations and the intervention that produced them, and the resulting Gaussian belief is trained to predict clinicians' next interventions, time to the next episode, and next-episode mortality. The authors release a benchmark derived from MIMIC-IV, MIMIC-IV-Note, and MIMIC-CXR covering 72,075 hospital admissions and 5.4M records organized into temporally ordered therapeutic-diagnostic episodes. Compared with an otherwise identical history-free model, OICycle improves next-intervention adaptive recall from 0.253 to 0.347, time-to-next-episode macro-F1 from 0.293 to 0.351, and next-episode mortality macro-F1 from 0.585 to 0.649.

Clinical AIPOMDPElectronic health records
Research Editor selected

SERIAL-7: A Benchmark for Machine-Verifiable Structured Generation

The paper introduces SERIAL-7, a Korean-English benchmark for evaluating machine-verifiable generation across seven representations. It separates semantic derivation, constrained artifact generation, and cross-representation preservation with 43 atomic constraints and executable checks. Results across ten models show that reliable structured generation requires explicit verification of how artifacts are realized and preserved.

Why it matters: Provides a concrete new benchmark and verification framework that addresses a critical gap in ensuring reliable structured outputs from LLMs.

large language modelsstructured outputverification
Research Editor selected

ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science

ProSearch is a new benchmark designed to evaluate multi-constraint protocol retrieval in experimental science, featuring 1,515 queries across six domains. It incorporates realism, complexity, and diagnosticity with 3-5 explicit constraints per query. Experiments with 19 retrievers show the best achieves nDCG@10 of 0.290, improving to 0.474 with reranking, identifying joint constraint binding as a key bottleneck.

Why it matters: Provides a systematic testbed for evaluating scientific retrieval systems under realistic multi-constraint conditions, highlighting current limitations and guiding future improvements.

ProSearchmulti-constraint retrievalexperimental science
Research Editor selected

BlindToChange: AI Agents Fail to Proactively Search for Recent Events

A new benchmark, BlindToChange, reveals that AI agents often fail to proactively search for recent real-world events that affect their responses, even though they can find changes when explicitly asked. The benchmark uses requests where an obvious candidate is invalidated by a recent event, and shows a performance gap of up to 27.5 points when the event is not mentioned. Over half of failures stem from agents not directly verifying relevant facts. This work highlights a new axis for progress in agentic systems.

Why it matters: Provides a concrete, measurable benchmark exposing a critical blind spot in AI agent behavior, directly relevant to improving real-world reliability.

AI agentsbenchmarkproactive information seeking