AWS sagemaker medium security documentation change
Summary
Added documentation for May 13, 2025 AMI release with Ubuntu 22.04 LTS support, including security updates, upgrade procedures, and compatibility guidance
Security assessment
The update explicitly states the AMI provides 'enhanced security through comprehensive package updates' and includes security-relevant components like Linux kernel 6.8 (up from 5.15), glibc 2.35 (up from 2.31), and NFS 2.6.1. These updates address potential vulnerabilities in older software versions and improve system hardening.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-release-ami-slurm.md b/sagemaker/latest/dg/sagemaker-hyperpod-release-ami-slurm.md index 507771c9f..a9bd5aee6 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-release-ami-slurm.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-release-ami-slurm.md @@ -5 +5 @@ -April 28, 2025February 18, 2025December 21, 2024November 24, 2024November 15, 2024November 11, 2024October 21, 2024September 10, 2024March 14, 2024November 29, 2023 +May 13, 2025April 28, 2025February 18, 2025December 21, 2024November 24, 2024November 15, 2024November 11, 2024October 21, 2024September 10, 2024March 14, 2024November 29, 2023 @@ -14,0 +15,104 @@ To update existing HyperPod clusters with the latest DLAMI, see [Update the Sage +## SageMaker HyperPod AMI releases for Slurm: May 13, 2025 + +Amazon SageMaker HyperPod released an updated AMI that supports Ubuntu 22.04 LTS for Slurm clusters. AWS regularly updates AMIs to ensure you have access to the most current software stack. Upgrading to the latest AMI provides enhanced security through comprehensive package updates, improved performance and stability for your workloads, and compatibility with new instance types and latest kernel features. + +###### In this release note, you will see: + + * Key updates in the Ubuntu 22.04 AMI + + * Upgrading to the Ubuntu 22.04 AMI + + * Troubleshooting upgrade failures + + + + +### Key updates in the Ubuntu 22.04 AMI + +The following table lists the component versions of the Ubuntu 22.04 AMI compared to the previous AMI. + +Component versions of the Ubuntu 22.04 AMI compared to the previous AMI Component | Previous version | Updated version +---|---|--- +**Ubuntu OS** | 20.04 LTS | 22.04 LTS +**Slurm** | 24.11 | 24.11 (unchanged) +**Python** | 3.8 (default) | 3.10 (default) +**Elastic Fabric Adapter (EFA) on Amazon FSx** | Not supported | Supported +**Linux kernel** | 5.15 | 6.8 +**GNU C Library (glibc)** | 2.31 | 2.35 +**GNU Compiler Collection (GCC)** | 9.4.0 | 11.4.0 +**libc6** | ≤ 2.31 | ≥ 2.35 supported +**Network File System (NFS)** | 1:1.3.4 | 1:2.6.1 + +###### Note + +Although the Slurm version (24.11) remains unchanged, the underlying OS and library updates in this AMI may affect your system behavior and workload compatibility. You must test your workloads before upgrading production clusters. + +### Upgrading to the Ubuntu 22.04 AMI + +Before upgrading your cluster to the Ubuntu 22.04 AMI, complete these preparation steps and review the upgrade requirements. To troubleshoot upgrade failures, see Troubleshooting upgrade failures. + +#### Review Python compatibility + +The Ubuntu 22.04 AMI uses Python 3.10 as the default version, upgraded from Python 3.8. Although Python 3.10 maintains compatibility with most Python 3.8 code, you should test your existing workloads before upgrading. If your workloads require Python 3.8, you can install it using the following command in your lifecycle script: + + + yum install python-3.8 + +Before upgrading your cluster, make sure to do the following: + + 1. Test your code compatibility with Python 3.10. + + 2. Verify your lifecycle scripts work in the new environment. + + 3. Check that all dependencies are compatible with the new Python version. + + 4. If you created your HyperPod cluster by copying the default lifecycle script from GitHub, add the following command to your `setup_mariadb_accounting.sh` file before upgrading to Ubuntu 22. For the complete script, see [setup_mariadb_accounting.sh on GitHub](https://github.com/aws-samples/awsome-distributed-training/blob/main/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/setup_mariadb_accounting.sh). + + apt-get -y -o DPkg::Lock::Timeout=120 update && apt-get -y -o DPkg::Lock::Timeout=120 install apg + + + + +#### Upgrade your Slurm cluster + +You can upgrade your Slurm cluster to use the new AMI in two ways: + + 1. Create a new cluster using the [`CreateCluster`](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCluster.html) API. + + 2. Update an existing cluster's software using the [`UpdateClusterSoftware`](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateClusterSoftware.html) API. + + + + +#### Validated configurations + +AWS has tested a wide range of distributed training workloads and infrastructure features on G5, G6, G6e, P4d, P5, and Trn1 instances, including: + + * Distributed training with PyTorch (e.g., FSDP, NeMo, LLaMA, MNIST). + + * Accelerator testing across instance types with Nvidia (P/G series) and AWS Neuron (Trn1). + + * Resiliency features that include [auto-resume](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm.html#sagemaker-hyperpod-resiliency-slurm-auto-resume) and [deep health checks](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-eks-resiliency-deep-health-checks.html). + + + + +#### Cluster downtime and availability + +During the upgrade process, the cluster will be unavailable. To minimize disruption, do the following: + + * Test the upgrade process on smaller clusters. + + * Create checkpoints before the upgrade, then restart training workloads from existing checkpoints after the upgrade completes. + + + + +### Troubleshooting upgrade failures + +When an upgrade fails, first determine if the failure is related to lifecycle scripts. These scripts commonly fail due to syntax errors, missing dependencies, or incorrect configurations. + +To investigate failures related to lifecycle scripts, check CloudWatch logs. All SageMaker HyperPod events and logs are stored under the log group: `/aws/sagemaker/Clusters/[ClusterName]/[ClusterID]`. Look specifically at the log stream `LifecycleConfig/[instance-group-name]/[instance-id]`, which provides detailed information about any errors during script execution. + +If the upgrade failure is unrelated to lifecycle scripts, collect relevant information including the cluster ARN, error logs, and timestamps, then contact [AWS support](https://aws.amazon.com/premiumsupport/) for further assistance. +