Live intelligence

Latest AI updates

2 updates
Tutorial

Route coding agent requests to the right model on Amazon Bedrock with NVIDIA NeMo Switchyard

AWS describes a deployable sample that routes coding-agent requests on Amazon Bedrock through LiteLLM and NVIDIA NeMo Switchyard. The sample offers Stage routing between a capable Claude model and a lower-cost Nemotron model, plus Random routing for A/B traffic splits, and uses AWS CDK to deploy LiteLLM on ECS Fargate. It includes benchmark runners and single-model baselines for measuring cost per solved task rather than relying on per-token pricing alone. The post explicitly labels the implementation as sample code for non-production use.

Amazon BedrockNVIDIA NeMo SwitchyardLiteLLM
Tutorial

Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

AWS describes how to build a self-service LLM gateway using Amazon Bedrock Application Inference Profiles and LiteLLM running on AWS Fargate, exposing an OpenAI-compatible API for developer tools. The post covers per-user API keys, spend budgets, rate limits, inference-profile routing, and observability with CloudWatch and Langfuse. It also notes that self-service key issuance requires LiteLLM Enterprise and recommends adding responsible AI controls before production. The architecture is presented with illustrative configuration snippets and AWS CDK constructs.

Amazon BedrockLiteLLMLLM gateway