Live intelligence

Latest AI updates

3 updates
Research

Study Finds Chatbot Arena Voters Grow Harsher Over Time, Even on Unchanged Models

Researchers analyzed 1,260,967 blind votes cast on Chatbot Arena between May 2023 and August 2024, restricting the analysis to pairs of models whose versions were fixed. For the same unchanged pair, the share of votes saying "both are bad" rose 1.71 percentage points per month against a mean of 17.1%, and 1.91 points per month in 2024. In fourteen pairs observed often in both years, the rate rose in every one, from 15.5% to 23.3%, while the stronger model's share of decisive votes stayed flat in 2024. The authors argue this reflects standards that track the category's progress, so absolute verdicts drift even as relative comparisons hold steady.

Chatbot ArenaLLM evaluationHuman feedback
Research Editor selected

Alibaba’s Co-Design Bench Exposes Gaps in AI Coding Agents

Alibaba introduced Co-Design Bench, a benchmark based on real user requests that coding agents previously failed in practice. The top-performing model, GPT-6 Astra, fully solved 26.4% of tasks, despite the same models scoring above 99% on older benchmarks. The results suggest established evaluations may substantially overstate coding-agent reliability in real-world workflows.

Why it matters: The benchmark offers useful evidence that current coding-agent scores may not reflect how much human cleanup practical tasks require.

Co-Design BenchAI coding agentsbenchmark reliability
Research

Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success

This arXiv paper proposes prefix executability, a trajectory-level evaluation protocol that checks whether every prefix of a predicted tool-call sequence stays executable under tool schemas, state preconditions, and data dependencies. It summarizes reliability with Prefix Executability Curves, First-Error Depth, Trajectory Executability Rate, horizon-normalized AUC-PEC, and Recovery Rate. The protocol is instantiated on stratified samples from BFCL-v3 multi-turn and ToolSandbox, plus the complete Airline/Retail base splits of the tau^3-bench. The authors report that outcome-only and prefix-level rankings diverge, with some plan-conditioned agents scoring higher final success while having much lower trajectory executability, and that explicit pre-execution plan checking can improve executable-prefix survival while keeping terminal success competitive.

tool-using agentsagent evaluationbenchmark reliability