What changed

Airbnb has implemented a system to evaluate and manage coding agents on Amazon Bedrock by using cost per task metrics. This approach helps guide model selection and monitor production usage as models and their usage patterns change. The company explains how cost per task, along with runtime signals, helps in diagnosing production issues and adjusting model defaults and effort ceilings.

What this means in practice

Airbnb's internal tool, AirChat, allows thousands of engineers to run coding agents weekly. These agents send model requests through an LLM gateway that runs on Amazon Bedrock.

The company runs evaluations (evals) on its own tasks to measure each model's score and cost per task before a rollout. After deployment, observability tools like Amazon CloudWatch help track changes in production.

Airbnb also uses cost per task to set default models and effort ceilings for coding tools, ensuring optimal performance and cost efficiency.

Limitations and availability

The supplied announcement does not specify the exact models used or the specific cost per task figures. It also does not mention the exact number of tasks evaluated or the specific regions where the models are available.

The company emphasizes that model releases can change output tokens, cost, latency, and capacity simultaneously, and that benchmark charts do not reflect how models handle their engineers' work.

Airbnb recommends running evaluations side by side with production data to ensure accurate model selection and performance monitoring.

Source facts for Airbnb uses cost per task to manage coding agents on Amazon Bedrock: It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS. As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: How Airbnb uses cost per task to operate coding agents on Amazon Bedrock