Live intelligence

Latest AI updates

22 updates
Products Editor selected

Airbnb Uses Cost per Task to Evaluate and Operate Coding Agents on Amazon Bedrock

AWS describes how Airbnb runs coding agents through an LLM gateway on Amazon Bedrock and evaluates candidate models on its own engineering tasks before rollout. Airbnb measures cost per task across uncached input, cache reads, cache writes and output tokens, then uses CloudWatch runtime metrics to investigate latency, throttling and spend in production. After one model scored well in evals, production usage showed much higher spend per user, driven largely by cached context and longer self-directed sessions, prompting a move to a cheaper model. Airbnb also uses evals to set reasoning-effort ceilings and to identify equivalent models for rebalancing traffic.

Why it matters: It offers a concrete, production-grounded playbook for selecting and monitoring coding-agent models by cost per task rather than benchmark scores alone.

Amazon BedrockAirbnbCoding agents
Models

Hugging Face Releases pi-Tuned LoRA Adapter for Qwen3.8-27B, Cutting Coding-Agent Token Use

Hugging Face's harness-hub team released Qwen3.8-27B-pi-adapted, an Apache-2.0 rank-32 LoRA adapter trained with AsyncGRPO to operate inside the pi coding agent. The team reports SWE-bench Pro (hard 51) pass@1 rising from 39.2% to 64.7% with thinking off, and median turns falling from 59 to 23 on tasks both models solved. The adapter matches the base model's thinking-mode SWE-bench Pro accuracy while using roughly 4.5x fewer tokens, though the results are self-reported on small evaluation sets and checkpoint selection partly used benchmark data. Deployment options include vLLM with LoRA enabled, a llama.cpp GGUF adapter, and a merged 54 GB BF16 checkpoint.

QwenHugging FaceLoRA
Products

External sandbox pattern released with three sandbox server implementations for coding agents

Simon Willison, working with Prime Radiant, introduced an external sandbox pattern in which code executed by an agent runs inside a dedicated sandbox while the agent itself stays outside it. Three implementations were released: external-sandbox-v86 (Node.js server running Alpine Linux via v86 in WebAssembly), external-sandbox-container (Python server using an Apple Containers instance), and external-sandbox-quicksand (Python server using Microsoft's Quicksand library with QEMU and Alpine Linux). All expose the same localhost API and can be configured for Pi, Claude Code, or Codex to route code execution to the sandbox. The sandboxes also include an optional HTTP/HTTPS allow-list proxy and a credential replacement mechanism for authenticated API calls without sharing secrets with the agent.

AI agent securitycode execution sandboxingcoding agents
Research

Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

Researchers introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task supplies a working program plus workload, correctness requirements, timing boundaries and permitted edits, with public testing for iteration and a separate verifier scoring the final submission. On version 1.1, 16 coding agents (model plus harness pairings) score between 25.5% and 67.4% Score@1, with most programs correct and speedup separating the leaders. The two leading agents reach at least 2x speedup on 22 or more of the 32 tasks, while the other fourteen hit 2x on only 8 to 17 tasks, leaving 125 of their 412 correct outcomes below 1.25x. Controlled replays show build and runtime settings alone can swing identical source code between 0.29x and 2.12x, highlighting the need for coordination across a full application.

coding agentsprogram optimizationbenchmarks
Products

Use prompt caching with open-weight models on Amazon Bedrock

Amazon Bedrock now supports explicit prompt caching for open-weight models, alongside implicit caching, to reduce costs and latency by reusing repeated prompt prefixes. The post demonstrates both modes via the Chat Completions API, including code examples and how to read cache usage fields. It also applies implicit caching to an OpenCode coding agent session and provides best practices.

Amazon BedrockPrompt CachingOpen-weight Models
Tutorial

Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

AWS describes a deployable sample that routes coding-agent requests on Amazon Bedrock through LiteLLM and NVIDIA NeMo Switchyard. The sample offers Stage routing between a capable Claude model and a lower-cost Nemotron model, plus Random routing for A/B traffic splits, and uses AWS CDK to deploy LiteLLM on ECS Fargate. It includes benchmark runners and single-model baselines for measuring cost per solved task rather than relying on per-token pricing alone. The post explicitly labels the implementation as sample code for non-production use.

Amazon BedrockNVIDIA NeMo SwitchyardLiteLLM
Research

Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments

The paper presents drydock-eval, a local evaluator for small coding agents that runs jobs in fresh network-isolated containers and checks local scoring behavior against a hosted scorer. It reports throughput of about 250 runs per hour for Gemma 4 E4B on one RTX 5090 and releases the full experimental record, including 9,660 agent runs, 2,494 grading controls and 72 serving measurements. Experiments found no detected E4B solve-rate change from four prompt divergences and no confirmation of three agent changes that had led their initial screens. The unchanged baseline also moved by 1.17 points when rerun, underscoring run-to-run variability in agent evaluation.

drydock-evalcoding agentsLLM evaluation
Research Editor selected

Microsoft Research Focus – Edition #98

This Microsoft Research podcast roundup highlights several concrete AI advances, including a resolved 31-year-old mathematics challenge known as the KLS conjecture, a new benchmark and method for making AI coding agents more efficient, and a tokenizer designed to preserve linguistic structure across 104 languages. It also covers progress toward AI-powered universal translation and a research internship experience. The episode bundles multiple verified research developments from Microsoft.

Why it matters: Readers get a compact overview of several recent, concrete Microsoft research breakthroughs in AI and mathematics, with specific tools and results.

AI researchmathematicscoding agents
Research Editor selected

CAMP-II trains coding agents to use files as long-horizon memory

Researchers introduce CAMG-II, a suite of long-horizon agentic-RL environments spanning Shop, Coding, AutoResearch, General, and Visual tasks, each with Bash access and an episode-persistent workspace. They also present CAMP-II, which trains a single policy across all five environments using fully asynchronous on-policy distillation from a frozen teacher with prior file-operation experience. On CAMG-II, CAMP-II reports 2.97 times the gain of the strongest OPD baseline. The learned memory behavior also transfers to Terminal-Bench 2.1 and SWE-Bench Pro V2 with improved success rates.

Why it matters: It provides a concrete benchmark and training approach for file-based memory in long-horizon coding agents, with reported gains and transfer to external agent benchmarks.

coding agentsagent memoryon-policy distillation
Research Editor selected

Embodied First Exam benchmark tests coding agents on robot control

The paper introduces Embodied First Exam, a unified agent-native benchmark for coding agents that control robots. It covers 1,156 tasks across 35 suites and 47 robots, including new suites for spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering. Agents can choose direct control, code-to-control, or policy training, with Standard, Realtime, and Privileged evaluation settings. On 456 core tasks, the strongest of eight agents, GPT-6 Astra in Codex, reaches a 34% success rate in the Standard Setting.

Why it matters: It offers a concrete, broad benchmark and baseline results for assessing whether coding agents can reliably operate robots under physical-world constraints.

Embodied First Examcoding agentsrobotics benchmarks
Research

When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents

The paper studies how stale repository instruction files such as AGENTS.md and CLAUDE.md affect LLM-based coding agents. Using three real-world stale instructions, 22 maintenance tasks, and 732 trajectories from two models, it finds agents follow stale guidance when it is relevant to their tasks, sometimes even when correct information is available elsewhere. On a stale test command, Claude Sonnet 5.5 needed about two extra tool calls per run compared with the corrected command, while no detectable difference appeared in the other cases. Agents generally recovered from immediate failures but did not repair the underlying instruction files.

coding agentsdocumentation debtLLM agent evaluation
Industry Editor selected

How Stanford Teaches AI-Native Engineering — Mihail Eric, Monaco

Stanford's CS146S course, taught by Mihail Eric, now requires students to contribute to unfamiliar open-source repos and have their patches reviewed by maintainers, reflecting real-world AI agent workflows. The interview covers the course syllabus, Monaco's agent workspaces, and the decision framework for buying vs. building developer tools. It also explores how coding agents change code review, on-call loops, and the role of junior engineers.

Why it matters: Provides concrete, actionable insights from a leading AI engineering educator and practitioner on adapting software engineering education and tooling for the age of coding agents.

AI engineering educationcoding agentssoftware development workflow
Research

Terminal-Bench 2.1: Continuous Benchmark Maintenance Reveals Stale Evaluations

The paper introduces the concept of benchmark maintenance debt, where benchmarks become stale as models improve and infrastructure changes. The authors propose a continuous-validation workflow and apply it to Terminal-Bench 2.0, finding issues in roughly 30% of the benchmark and releasing repairs as Terminal-Bench 2.1. These repairs change evaluation results, with accuracy increasing by over 10 percentage points and leaderboard positions shifting by up to three ranks, while overall ranking remains highly correlated.

benchmark maintenanceLLM evaluationcoding agents
Research

Study Finds Coding Agents Forget Previous Tasks During Continual Harness Evolution

A new empirical study investigates whether coding agent harnesses can evolve continuously across multiple tasks without catastrophic forgetting. Using six evolution directions across three software engineering datasetshare, the authors show that evolving on a new dataset can severely degrade performance on previously learned tasks—e.g., Meta-Harness drops from 94.0% to 32.0% on SWE-Bench after five rounds of further evolution. The paper provides insights for designing continual self-evolution methods for agent harnesses.

agent self-evolutioncontinual learningcoding agents
Products Editor selected

Factory Releases Droid 1.0 With Programmatic Tool Calling Enabled by Default

Factory has released Droid 1.0, a model-independent coding agent harness that enables programmatic tool calling by default. Its Script tool runs JavaScript in a sandbox, coordinating multiple tool calls while returning only printed output to the model to limit context usage. The update also reduces the base prompt and loads tools on demand; Factory reports performance and cost improvements, but the cited benchmark results are self-reported.

Why it matters: The release offers developers a concrete approach to reducing coding-agent context overhead and coordinating tools across supported models.

FactoryDroidCoding agents
Tutorial

Amazon Bedrock AgentCore runtime adds session endpoints for live web app previews

AWS introduced session endpoints for Amazon Bedrock AgentCore runtime, giving each agent session an authenticated URL that forwards HTTP/1.1, HTTP/2 and WebSocket traffic to ports the developer explicitly allows. The post walks through a sample coding agent that keeps a Vite dev server running inside its microVM session, so developers can watch a site update in the browser as the agent saves files. It covers port configuration, SigV4 or JWT authentication, a local-endpoint proxy for browser clients, and the specific 403, 404 and 502 refusal codes the runtime returns.

Amazon Bedrock AgentCorecoding agentsAWS
Products Editor selected

Microsoft and UC Berkeley Release TokenEconomizer to Cut Coding Agent Costs by Up to 30.9%

Microsoft Research and UC Berkeley researchers released TokenEconomizer, an Apache 2.0 open-source cost controller that plugs into Claude Code, Codex, GitHub Copilot CLI and Pi without modifying model weights or agent code. It learns per-repository policies from past traces through three mechanisms: micro-skills, tool pruning and failure-loop intervention. Across 277 SWE-Bench tasks and five agent-model settings, it reduced mean cost by 18.4% to 30.9%, and it was cheaper in all 30 repository-setting cells. Pass@1 improved in four of five settings and fell 1.0 point on Claude Code with Opus 4.8.

Why it matters: It gives teams a concrete, installable way to lower coding-agent token spend, backed by per-setting SWE-Bench numbers and an honest account of where savings are smaller.

Microsoft Researchcoding agentstoken efficiency
Research

Gym-Anything for Robotics uses coding agents to scale robotics simulation creation

The paper proposes Gym-Anything for Robotics, a framework in which coding agents build realistic robotics simulations, on the premise that simulation creation is itself a coding task that improves as coding agents improve. The authors select simulation domains based on U.S. GDP data and define criteria for good simulations and tasks, then apply an adversarial creation-audit loop. The resulting RoboWorld suite contains 25 simulations spanning 25 test and 175 training RL environments of varying difficulty. Frontier models reportedly average only 0.06 reward on RoboWorld test tasks, and the paper introduces in-context learning over repeated attempts, a privileged teacher-student setup, self-assisted learning, and code policies that generate demonstrations for training VLAs such as pi_0.5.

robotics simulationcoding agentsembodied AI