Live intelligence

Latest AI updates

1 updates
Products Editor selected

Airbnb Uses Cost per Task to Evaluate and Operate Coding Agents on Amazon Bedrock

AWS describes how Airbnb runs coding agents through an LLM gateway on Amazon Bedrock and evaluates candidate models on its own engineering tasks before rollout. Airbnb measures cost per task across uncached input, cache reads, cache writes and output tokens, then uses CloudWatch runtime metrics to investigate latency, throttling and spend in production. After one model scored well in evals, production usage showed much higher spend per user, driven largely by cached context and longer self-directed sessions, prompting a move to a cheaper model. Airbnb also uses evals to set reasoning-effort ceilings and to identify equivalent models for rebalancing traffic.

Why it matters: It offers a concrete, production-grounded playbook for selecting and monitoring coding-agent models by cost per task rather than benchmark scores alone.

Amazon BedrockAirbnbCoding agents