Live intelligence

Latest AI updates

10 updates

Research

No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark

SemiAnalysis has introduced a new benchmark called GDPx that evaluates AI agents on simulated knowledge work tasks, spanning domains like chip design and quant trading. According to the benchmark, no model achieves more than 12 percent, with Claude Opus 5.5 leading slightly ahead of Gemini 4 Carbon. Critics note the benchmark has a small number of tasks per domain and uneven testing conditions.

the-decoder5h agoScore 70

AI benchmarkingAI agentsSemiAnalysis

Research Editor selected

Fairness Metrics based on the AUC in Clinical Predictive AI

This paper examines eight AUC-based fairness metrics for clinical prediction models, highlighting that the AUC is non-collapsible, so subgroup and overall values are not directly comparable. The authors characterize each metric's estimand, relate them to the decomposition of overall AUC, and use EuroSCORE II simulations to illustrate behavior. They recommend within-group AUC parity and caution against gap-to-overall, equity-scaled, and minimum-based metrics.

Why it matters: Provides a critical evaluation of fairness metrics with practical recommendations for clinical AI, directly useful for model developers and researchers.

arxiv1d agoScore 82

fairness in AIclinical predictive modelsAUC decomposition

Research Editor selected

From the Zoo to the BULLPEN: Sample-Efficient LLM Profiling via Multi-Channel Model Embeddings

BULLPEN is a new framework that extends EmbedLLM to profile large language models by jointly encoding answer text, correctness, and question features. It adds 20 behavioral, style, and provenance benchmarks and achieves higher accuracy on these tasks using only a few hundred questions instead of the full 24,118-question pool. The method retains at least 91% of full-pool skill with just 400 questions, enabling low-cost, sample-efficient profiling that works even for API-only models.

Why it matters: Provides a practical, sample-efficient method for profiling LLM behaviors (verbosity, safety, style) that reduces inference cost by over 98%, valuable for model selection and auditing.

arxiv3d agoScore 85

large language modelsmodel evaluationembedding methods

Research Editor selected

Simple Features Outperform Most AI Control Monitors on Standard Dataset

A new arXiv paper shows that simple stylistic and problem-level features can outperform 20 of 24 open-weight AI control monitors on a standard backdoor detection dataset, without executing code. The dataset leaks labels through author style and problem characteristics, and one in seven backdoors fails validity checks, distorting comparisons. The authors release a rebuilt dataset and checks, cautioning that high scores on such benchmarks do not prove sabotage detection.

Why it matters: This research exposes critical flaws in AI safety evaluation benchmarks, offering actionable insights for improving control monitors and dataset design.

arxiv3d agoScore 85

AI safetybenchmark designmodel evaluation

Research Editor selected

CRoS: A New Causality Reward for More Faithful Chain-of-Thought Reasoning

Researchers propose CRoS, a causality metric that perturbs the order of reasoning steps without altering content, avoiding confounding effects in existing perturbation-based metrics. On Reasoning Gym with Gemma-3-4B, CRoS improves out-of-domain accuracy, CoT quality, and faithfulness compared to existing rewards. It achieves 46.8% accuracy on unseen datasets, outperforming the strongest baseline by 3 percentage points.

Why it matters: This introduces a novel method to improve reasoning faithfulness in RL-trained models, with concrete benchmark gains, valuable for AI researchers and practitioners.

arxiv3d agoScore 82

Chain-of-Thought ReasoningReinforcement LearningCausality Metrics

Research

Knowledge Layer Model: A Spec for Evidence-Visible AI Output with Mixed Results

The paper introduces the Knowledge Layer Model (KLM), an open specification for inference records with epistemic assessments and evidence relations, implemented in the Nage AI OS as SEDIM. It reports internal experiments: a routing benchmark achieved 91.3% accuracy, but diagnostics failed to distinguish silent from contradicted examples (AUC 0.008), and a native-model configuration produced 0/15 successful answers. The authors emphasize that evidence visibility does not guarantee correctness and call for independent replication.

arxiv3d agoScore 72

AI transparencyinference recordsmodel evaluation

Research

GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains

GDPx is described as a frontier benchmark spanning domains including RSI, hardware design, cyber, bio, finance, legal, quant trading, chip design, software engineering, materials science, and alignment. The supplied material provides only a high-level description, without methodology, results, or model comparisons. It is presented as a way to measure frontier model capabilities across economically valuable domains.

AIHUB3d agoScore 72

AI benchmarksfrontier modelsmodel evaluation

Research

Study: Chinese AI models show political complexity, not uniform government alignment

A study by Vals AI tested 25 AI models for alignment with Chinese government positions and found that Chinese models respond less uniformly than expected. Western models occasionally adopt official Beijing positions, and the language of the prompt strongly influences political slant. The findings challenge the stereotype of Chinese AI as mere government mouthpieces.

The Decoder4d agoScore 72

AI alignmentpolitical biasChinese AI models

Tutorial Editor selected

Fixing evaluation bias for model selection on Amazon Bedrock

AWS and Property Finder describe how 96.8% of evaluation labels had been inherited from the incumbent model, biasing comparisons against alternatives. They built expert-annotated ground truth and used iterative prompt optimization to select Llama 4 Maverick 17B Instruct on Amazon Bedrock. The post reports development-set Micro F1 rising from 0.55 to 0.81 and an estimated 86% cost reduction, with a caveat that evaluation-set scores are optimistic because prompts were tuned on included images.

Why it matters: It gives practitioners a concrete methodology for auditing biased evaluation labels and running fairer model selection, with reported accuracy and cost tradeoffs.

AWS Machine Learning BlogOct 28, 2026Score 88

Amazon Bedrockmodel evaluationprompt optimization

Research

Microsoft claims AI diagnostic tool outperforms doctors in Science study

Microsoft says it has built an AI-powered diagnostic tool that outperforms human doctors by up to four times on its own benchmark. The results were published in the journal Science and based on comparisons against physicians using scenarios drawn from medical case reports and emergency department data. Microsoft AI CEO Mustafa Suleyman framed the tool as healthcare AI 'finally here.' The benchmark was compiled by Microsoft itself, and the available text gives no further methodological detail.

The Verge AIOct 23, 2026Score 75

MicrosoftAI in healthcarediagnostic AI