AWS has introduced rate limiting capabilities for AI traffic on its AgentCore gateway, enabling organizations to manage how much traffic individual users can consume through the gateway. This feature allows for fine-grained control over traffic consumption, ensuring downstream services remain available during heavy traffic spikes. Users can define OAuth or IAM-based rules for requests per minute, concurrent connections, and token throughput. The gateway measures every incoming request against these limits, ensuring fair usage and preventing overloads. This update enhances the security and performance of AI workflows by providing centralized rate limiting for various target types, including MCP targets, inference targets, and HTTP passthrough targets. Source: awsml
Rate limiting in AgentCore gateway supports three main metrics: request rate limits, token rate limits, and connection rate limits. Request rate limits are measured in requests per second (RPS) and requests per minute (RPM), applying to all target types. Each limit defines a maximum count of requests permitted within the given time window, with every request counting as one unit toward the configured limit, regardless of its duration. Token rate limits, measured in tokens per minute (TPM), apply only to inference targets and account for both input and output tokens. Connection rate limits, measured in connections per second (CPS), apply to all target types and track how long each request holds an open connection, especially for long-lived sessions. Source: awsml
The configuration of rate limits in AgentCore gateway involves dimension keys and entries, where dimension keys define how the gateway groups incoming traffic into rate buckets, and entries define the allowed throughput for each bucket. AWS provides examples of how to use the AWS CLI to create rate limit configurations, demonstrating the relationship between dimension keys and entries. The gateway supports various dimension keys, such as targetName, toolName, and qualifiedModelId, allowing for granular control over traffic based on specific criteria. Source: awsml
