AWS wellarchitected documentation change
Summary
Expanded documentation on fault tolerance mechanisms in SageMaker AI HyperPod, including EKS/Slurm implementations, checkpointing strategies, and managed service recommendations. Restructured implementation steps to emphasize managed services and monitoring.
Security assessment
The changes focus on improving fault tolerance and reliability of distributed training workloads through checkpointing, node recovery, and managed services. While these enhance system resilience, there is no direct mention of security vulnerabilities, access controls, or data protection mechanisms. The content addresses operational reliability rather than security threats.
Diff
diff --git a/wellarchitected/latest/generative-ai-lens/genrel06-bp01.md b/wellarchitected/latest/generative-ai-lens/genrel06-bp01.md index f361be0b6..08e15a719 100644 --- a//wellarchitected/latest/generative-ai-lens/genrel06-bp01.md +++ b//wellarchitected/latest/generative-ai-lens/genrel06-bp01.md @@ -22,0 +23,6 @@ Amazon SageMaker AI HyperPod clusters allow customers to pre-train or fine-tune +Amazon SageMaker AI HyperPod, with both Amazon EKS and Slurm orchestration, establishes comprehensive checkpointing mechanisms that automatically save training state at regular intervals to persistent storage like Amazon S3 or FSx for Lustre. + +For EKS-based HyperPod, use fault tolerance capabilities by implementing application-level checkpointing in your training scripts, and store checkpoints on shared persistent volumes that survive pod restarts and node failures. Configure Kubernetes health checks and restart policies to automatically detect and recover from failed training pods while preserving progress from the last checkpoint. + +For Slurm-based HyperPod, use the auto-resume functionality to provide zero-touch resiliency infrastructure that automatically recovers training jobs from the last saved checkpoint when hardware failures occur. Configure your training jobs to run inside exclusive allocations using salloc or sbatch, and verify that your entrypoint scripts maintain environment consistency across node replacements. Both systems benefit from SageMaker AI HyperPod's built-in cluster health monitoring that continuously checks GPU health with DCGM policies, network connectivity with EFA health checks, and automatically replaces faulty nodes. The multi-head node support in Slurm further enhances fault tolerance by providing backup head nodes that automatically take over if the primary head node fails. + @@ -25 +31,3 @@ When implementing fault-tolerant distributed training manually, evaluate options -### Implementation steps +Use managed services and purpose-built infrastructure to handle the complexity and resource requirements of distributed model customization workloads. AWS offers several solutions that can help improve the reliability and efficiency of these tasks: + + * **Amazon SageMaker AI HyperPod:** A managed service that automates the provisioning and orchestration of distributed training infrastructure, including handling node failures, checkpointing, and other fault-tolerance mechanisms. HyperPod is optimized for large language model training and can use specialized hardware like AWS Trainium instances. @@ -27 +35 @@ When implementing fault-tolerant distributed training manually, evaluate options - 1. In Amazon Bedrock, when using custom models: + * **Amazon Bedrock:** Provides managed workflows for fine-tuning, continued pre-training, and model distillation, abstracting away the underlying infrastructure management and failure handling. @@ -29 +37 @@ When implementing fault-tolerant distributed training manually, evaluate options - * Select a model customization job like fine-tuning or continued pre-training. + * **AWS Batch:** A fully-managed batch processing service that can run distributed computational tasks, including model customization, with automatic scaling, retry logic, and resource optimization. @@ -31 +38,0 @@ When implementing fault-tolerant distributed training manually, evaluate options - * Follow the prompts to begin executing the job. @@ -33 +39,0 @@ When implementing fault-tolerant distributed training manually, evaluate options - * Test the output once the job has completed. @@ -35 +40,0 @@ When implementing fault-tolerant distributed training manually, evaluate options - 2. Alternatively, provision SageMaker AI HyperPod or EC2 UltraClusters. @@ -37 +42,7 @@ When implementing fault-tolerant distributed training manually, evaluate options - 3. Configure object store for workload checkpointing. +When implementing fault tolerance manually, focus on strategies like checkpointing, progress tracking, and automated recovery. Use high-performance storage solutions like Amazon FSx for Lustre to provide rapid access to training data. Configure your workflow to handle node failures, spot instance interruptions, and other disruptions gracefully. + +Continuously monitor the distributed workloads for performance, resource utilization, and failures. Use Amazon CloudWatch to set alerts and thresholds, and use Amazon EventBridge to run automated remediation actions. Analyze logs and metrics to identify bottlenecks and optimize the distributed architecture over time. + +### Implementation steps + + 1. Evaluate managed services like SageMaker AI HyperPod, Bedrock, and Batch for your model customization needs. @@ -39 +50,7 @@ When implementing fault-tolerant distributed training manually, evaluate options - 4. Provision high performance Amazon FSx for Lustre containing your training and customization data. + 2. If implementing a custom distributed workflow, provision high-performance storage and compute resources. + + 3. Implement checkpointing, progress tracking, and automated retry mechanisms to handle failures. + + 4. Configure monitoring, alerting, and automated remediation for the distributed workloads. + + 5. Continuously analyze performance, costs, and reliability to optimize the distributed architecture. @@ -46 +63 @@ When implementing fault-tolerant distributed training manually, evaluate options -**Related practices:** +**Related best practices:** @@ -57 +74 @@ When implementing fault-tolerant distributed training manually, evaluate options -**Related guides, videos, and documentation:** +**Related documents:** @@ -62,0 +80,2 @@ When implementing fault-tolerant distributed training manually, evaluate options + * [Resilience-related Kubernetes labels by SageMaker AI HyperPod](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-resiliency-node-labels.html) + @@ -77,0 +97,8 @@ When implementing fault-tolerant distributed training manually, evaluate options + * [Introducing Amazon SageMaker AI HyperPod, a purpose-built infrastructure for distributed training at scale](https://aws.amazon.com/blogs/aws/introducing-amazon-sagemaker-hyperpod-a-purpose-built-infrastructure-for-distributed-training-at-scale/) + + * [Ray jobs on Amazon SageMaker AI HyperPod: scalable and resilient distributed AI](https://aws.amazon.com/blogs/machine-learning/ray-jobs-on-amazon-sagemaker-hyperpod-scalable-and-resilient-distributed-ai/) + + * [SageMaker AI HyperPod cluster resiliency](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm.html) + + * [Reduce ML training costs with Amazon SageMaker AI HyperPod](https://aws.amazon.com/blogs/machine-learning/reduce-ml-training-costs-with-amazon-sagemaker-hyperpod/) +