AWS wellarchitected high security documentation change
Summary
Expanded best practice guidance for mitigating system overload in generative AI workloads, adding scaling solutions and detailed implementation strategies for SageMaker HyperPod
Security assessment
The change explicitly adds rate limiting/throttling as a mitigation for denial-of-service attacks and discusses security benefits like enhanced protection against traffic spikes. The risk level increased from Medium to High, indicating greater security implications.
Diff
diff --git a/wellarchitected/latest/generative-ai-lens/genops02-bp03.md b/wellarchitected/latest/generative-ai-lens/genops02-bp03.md index 8e61f1d1a..165d5f3c9 100644 --- a//wellarchitected/latest/generative-ai-lens/genops02-bp03.md +++ b//wellarchitected/latest/generative-ai-lens/genops02-bp03.md @@ -7 +7 @@ Implementation guidanceResources -# GENOPS02-BP03 Implement rate limiting and throttling to mitigate the risk of system overload +# GENOPS02-BP03 Implement solutions to mitigate the risk of system overload @@ -9 +9,5 @@ Implementation guidanceResources -Implement rate limiting and throttling for AI application stability and performance. These practices control request processing rates to prevent system overload, which provides consistent application health and a better user experience. By adopting these measures, you can achieve balanced workload distribution, reduce service disruption risks, and enhance application reliability. This approach safeguards against excessive demand, optimizes resource utilization, and improves cost efficiency and performance. +There are two primary ways to mitigate the risk of system overload for generative AI workloads. The first is to scale the inference serving architecture using advanced auto-scaling technologies. This is possible using Amazon SageMaker AI Inference Components, which you can use to host and scale model independent of the underlying infrastructure. For self-hosted language models, this is the ideal approach. + +The second approach is to rate limit and throttle managed inference to maintain application stability and performance. This approach is more applicable to managed inference on Amazon Bedrock. This practice controls request processing rates to avoid system overload, which provides consistent application health and a better user experience. You can increase system throughput by opting for cross-Region inference or in some cases by purchasing provisioned model thoughput. + +By adopting these measures, you can achieve balanced workload distribution, reduce service disruption risks, and enhance application reliability. This approach safeguards against excessive demand, optimizes resource utilization, and improves cost efficiency and performance. @@ -22 +26 @@ Implement rate limiting and throttling for AI application stability and performa -**Level of risk exposed if this best practice is not established:** Medium +**Level of risk exposed if this best practice is not established:** High @@ -26 +30,3 @@ Implement rate limiting and throttling for AI application stability and performa -Implementing rate limiting and throttling is crucial for the stability of generative AI applications. This practice controls incoming request rates to reduce the risk of system overload, helping to provide consistent performance and availability. It protects against traffic spikes, can act as one of the mitigations to denial-of-service attacks, and promotes fair usage. Benefits include reliable performance, enhanced security, optimized resource utilization, and improved user experience, which align with key principles of reliability, performance efficiency, security, and cost optimization. +For self-hosted models, adopt SageMaker AI Inference Components. Inference Components are an extension of multimodel endpoints, and are meant for hosting and scaling large-language models dynamically. Inference components treat models as primary elements, scaling the underlying hardware as needed based on the availability of CPU and GPU resources, as well as the full inference load on the provisioned infrastructure. Inference components are meant for workloads where you have control over the underlying infrastructure, and therefore should not be considered for generative AI workloads hosted on managed infrastructure such as Amazon Q for Business or Amazon Bedrock. + +Implementing rate limiting and throttling is crucial for the stability of generative AI applications. This practice controls incoming request rates to reduce the risk of system overload, helping to provide consistent performance and availability. It helps protect against traffic spikes, can act as one of the mitigations to denial-of-service attacks, and promotes fair usage. Benefits include reliable performance, enhanced security, optimized resource utilization, and improved user experience, which align with key principles of reliability, performance efficiency, security, and cost optimization. @@ -29,0 +36,12 @@ When designing generative AI systems, consider the limitations of source systems +In SageMaker AI HyperPod with both Amazon EKS and Slurm orchestration, establish comprehensive request rate controls and resource throttling mechanisms that help protect your cluster from overload conditions while maintaining optimal training performance. + +For EKS-based HyperPod, implement rate limiting through managed Kubernetes orchestration with resource quotas and limit ranges to control resource consumption at namespace and pod levels, avoiding system overload during peak demand. Configure HyperPod Task Governance with intelligent throttling mechanisms that automatically manage task queues and resource allocation rates, verifying that production workloads receive priority processing while development tasks are throttled appropriately to avoid cluster saturation. + +Use horizontal pod autoscaling with conservative scaling policies and priority classes to implement request throttling based on workload criticality, while using node selectors to distribute load across different instance types and reduce hotspots. The usage reporting feature provides real-time visibility into resource consumption patterns, enabling proactive rate limiting adjustments based on GPU, CPU, and Neuron Core utilization metrics to maintain optimal cluster performance under varying load conditions. + +For Slurm-based HyperPod, use Slurm's native job submission throttling and fair share scheduling to avoid system overload by controlling the rate at which jobs are admitted to the cluster based on available resources and current system load. Implement quality of service (QoS) policies and job priority classes that automatically throttle lower-priority workloads when system resources approach capacity limits, while maintaining consistent processing rates for critical training jobs. + +Configure resource allocation policies that dynamically adjust job submission rates based on cluster health metrics, combined with HyperPod's auto-resume functionality to handle temporary overload conditions gracefully without cascading failures. + +Both systems benefit from implementing circuit breaker patterns through SageMaker AI HyperPod Recipes that provide pre-configured throttling mechanisms and rate limiting strategies optimized for specific model architectures like Llama and Mistral, providing sustained performance while reducing resource exhaustion and system instability during high-demand periods. + @@ -77 +95 @@ The embedding model has important performance considerations in your application -**Related practices:** +**Related best practices:** @@ -86 +104 @@ The embedding model has important performance considerations in your application -**Related guides, videos, and documentation:** +**Related documents:** @@ -98,0 +117,2 @@ The embedding model has important performance considerations in your application + * [Supercharge your auto scaling for generative AI inference – Introducing Container Caching in SageMaker AI Inference](https://aws.amazon.com/blogs/machine-learning/supercharge-your-auto-scaling-for-generative-ai-inference-introducing-container-caching-in-sagemaker-inference/) + @@ -104,0 +125,12 @@ The embedding model has important performance considerations in your application + * [Analyze Amazon SageMaker AI spend and determine cost optimization opportunities based on usage, Part 1](https://aws.amazon.com/blogs/machine-learning/part-1-analyze-amazon-sagemaker-spend-and-determine-cost-optimization-opportunities-based-on-usage-part-1/) + + * [Maximize Accelerator Utilization for Model Development with New Amazon SageMaker AI HyperPod Task Governance](https://aws.amazon.com/blogs/aws/maximize-accelerator-utilization-for-model-development-with-new-amazon-sagemaker-hyperpod-task-governance/) + + * [Introducing Amazon SageMaker AI HyperPod to train foundation models at scale](https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-to-train-foundation-models-at-scale/) + + * [Best practices for Amazon SageMaker AI HyperPod task governance](https://aws.amazon.com/blogs/machine-learning/best-practices-for-amazon-sagemaker-hyperpod-task-governance/) + + * [Get started with Amazon SageMaker AI HyperPod task governance](https://www.youtube.com/watch?v=_wDhBAPwhoM) + + * [Usage reporting for cost attribution in SageMaker AI HyperPod](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-usage-reporting.html) +