AWS wellarchitected documentation change
Summary
Enhanced guidance on throughput management for generative AI workloads with added details about monitoring, scaling strategies, request buffering, and quota management
Security assessment
The changes focus on performance optimization and reliability improvements through better scaling and throughput management. While rate limiting and quota management are mentioned, they are presented as capacity planning measures rather than security controls. No specific security vulnerabilities or protections are addressed.
Diff
diff --git a/wellarchitected/latest/generative-ai-lens/genrel01-bp01.md b/wellarchitected/latest/generative-ai-lens/genrel01-bp01.md index dcdb7baaf..2e3374da3 100644 --- a//wellarchitected/latest/generative-ai-lens/genrel01-bp01.md +++ b//wellarchitected/latest/generative-ai-lens/genrel01-bp01.md @@ -9 +9 @@ Implementation guidanceResources -Collect information on the generative AI workload's utilization. Use this information to determine the required throughput for your foundation model. +Collect information on the generative AI workload's utilization, and implement dynamic scaling strategies to match capacity with demand. Use this information to determine the required throughput for your foundation model and establish appropriate quotas and scaling policies. @@ -11 +11 @@ Collect information on the generative AI workload's utilization. Use this inform -**Desired outcome:** When implemented, this best practice improves the reliability of your generative AI workload by matching the configured or provisioned throughput to your foundation models to the workload's demand. +**Desired outcome:** When implemented, this best practice improves the reliability of your generative AI workload by matching the configured or provisioned throughput to your foundation models to the workload's demand. This results in optimal resource utilization and consistent performance under varying loads. @@ -19 +19 @@ Collect information on the generative AI workload's utilization. Use this inform -Foundation models have throughput quotas. Inference requests require significant computation and memory to serve, and latency may increase during periods of high inference demand, especially when model endpoints serve inference from multiple requests simultaneously. +When managing throughput for foundation models, consider implementing a comprehensive monitoring and scaling strategy. Use a robust monitoring system that provides detailed insights for tracking throughput metrics and creating alarms for quota utilization. @@ -21 +21 @@ Foundation models have throughput quotas. Inference requests require significant -For model endpoints hosted on Amazon Bedrock, consider provisioned throughput endpoints or cross-region inference profiles. Provisioned throughput provides dedicated infrastructure that can achieve higher, more stable throughput than allowed through default quotas for on demand models hosted on Amazon Bedrock. Provisioned throughput capacity can be monitored in Amazon CloudWatch, which helps you proactively scale when capacity nears critical thresholds. Cross-region inference profiles distribute inference demand over a region of availability. For model endpoints hosted on Amazon SageMaker AI Inference Endpoints, consider leveraging traditional throughput scaling techniques like EC2 Auto-Scaling groups behind a load balancer. If your increased throughput needs are periodic and predictable, consider deploying larger instance types in advance of the increased need. Ultimately, it is encouraged to proactively engage with AWS support to increase service quotas based on known workload demands. +To handle traffic spikes and maintain consistent performance, implement request buffering using a message queue service, which can help smooth out irregular traffic patterns and avoid overwhelming the model endpoints. Use a service quota management system to adjust service limits based on your workload requirements, while implementing auto-scaling mechanisms to enable dynamic capacity management based on demand. @@ -23 +23,7 @@ For model endpoints hosted on Amazon Bedrock, consider provisioned throughput en -Queuing is a powerful technique for consuming requests. Consider placing queues between generative AI applications and models so that models do not deny or drop requests due to throughput constraints. This architecture lends itself to event-driven messaging patterns, making it a particularly robust option for architectures with high demand. +Consider placing queues between generative AI applications and models so that models do not deny or drop requests due to throughput constraints. This architecture lends itself to event-driven messaging patterns, making it a particularly robust option for architectures with high demand. + +For handling common throughput bottlenecks, consider implementing token bucket algorithms for rate limiting or using provisioned throughput options when dealing with token rate limits. To address concurrent request limits, implement request queuing or distribute requests across multiple Regions. For model loading overhead, maintain a warm pool of model instances or implement model caching strategies. Each of these solutions should be monitored for effectiveness using your chosen metrics and monitoring system. + +Provisioned Throughput endpoints or cross-Region inference profiles on Amazon Bedrock may help to alleviate scaling bottlenecks for fully-managed inference hosting. Provisioned Throughput provides dedicated infrastructure that can achieve higher, more stable throughput than allowed through default quotas for on demand models hosted on Amazon Bedrock. Provisioned Throughput capacity can be monitored in Amazon CloudWatch, which helps you proactively scale when capacity nears critical thresholds. + +Cross-Region inference profiles distribute inference demand over a region of availability. For model endpoints hosted on Amazon SageMaker AI Inference Endpoints, consider using traditional throughput scaling techniques like EC2 Autoscaling groups behind a load balancer. If your increased throughput needs are periodic and predictable, consider deploying larger instance types in advance of the increased need. Ultimately, it is encouraged to proactively engage with AWS support to increase service quotas based on known workload demands. @@ -27 +33,13 @@ Queuing is a powerful technique for consuming requests. Consider placing queues - 1. Determine the foundation model that handles inference requests for your generative AI workload. + 1. Set up comprehensive monitoring using CloudWatch: + + * Create custom dashboards for throughput metrics + + * Configure alarms for quota utilization + + * Enable detailed monitoring for critical resources + + 2. Implement request management: + + * Deploy queue-based architecture for request buffering + + * Set up rate limiting at the application layer @@ -29 +47 @@ Queuing is a powerful technique for consuming requests. Consider placing queues - 2. Perform load testing on the workload to get a baseline of performance, identifying if an upper-limit on throughput is feasible for this application. + * Configure retry mechanisms with exponential backoff @@ -31 +49 @@ Queuing is a powerful technique for consuming requests. Consider placing queues - 3. Determine if cross-region inference profiles (if available for this model) increases the throughput. + 3. Configure scaling mechanisms: @@ -33 +51,13 @@ Queuing is a powerful technique for consuming requests. Consider placing queues - 4. Consider purchasing provisioned throughput if necessary. + * Set up auto-scaling policies based on demand + + * Configure provisioned throughput where appropriate + + * Implement cross-region request distribution + + 4. Establish ongoing optimization: + + * Regular review of utilization patterns + + * Periodic adjustment of quotas and scaling parameters + + * Continuous monitoring and refinement of thresholds @@ -61,0 +92,4 @@ Queuing is a powerful technique for consuming requests. Consider placing queues + * [Enable Amazon Bedrock cross-Region inference in multi-account environments](https://aws.amazon.com/blogs/machine-learning/enable-amazon-bedrock-cross-region-inference-in-multi-account-environments/) + + * [Building well-architected serverless applications: Regulating inbound request rates – part 1](https://aws.amazon.com/blogs/compute/building-well-architected-serverless-applications-regulating-inbound-request-rates-part-1/) +