What changed
AWS announced a new approach for developers to access foundation models (FMs) on Amazon Bedrock through a self-service LLM gateway built with LiteLLM. This solution enables developers to interact with FMs via an OpenAI-compatible API while providing centralized visibility, cost controls, and access management.
What this means in practice
Developers can configure their tools to point to the LLM gateway, which abstracts Amazon Bedrock behind an OpenAI-compatible API. This allows developers to access models like Claude Sonnet and Claude Haiku without interacting with Bedrock directly. The gateway includes a self-service portal for managing API keys and spend policies, with LiteLLM Enterprise required for key issuance. The architecture uses AWS Fargate, Amazon RDS, Amazon ElastiCache, and AWS Secrets Manager to manage authentication, rate-limiting, and cost tracking. Developers can set up their tools with a single configuration, including an API key and model name, enabling them to swap models or add fallbacks without changing their tool configurations.
Limitations and availability
The LiteLLM AI Gateway is open source and suitable for small pilots but has limitations. The free tier supports up to five users, and the self-service key issuance described requires the Enterprise version. The solution uses Amazon Bedrock Application Inference Profiles, which are available in specific AWS regions. The architecture includes components like Langfuse for observability and Amazon CloudWatch for infrastructure metrics. The supplied announcement does not specify exact availability timelines or additional supported models beyond the examples provided.
