AWS sagemaker documentation change
Summary
Added documentation for new AMI-based cluster configuration option without lifecycle scripts, including comparison of provisioning methods, extension scripts, and security implications of different configuration choices.
Security assessment
The change documents a new AMI-based configuration that eliminates internet access requirements during provisioning, reducing attack surface. It also clarifies security responsibilities when using custom scripts. While this improves security posture, there's no evidence of addressing a specific vulnerability.
Diff
diff --git a/sagemaker/latest/dg/smcluster-getting-started-slurm-console.md b/sagemaker/latest/dg/smcluster-getting-started-slurm-console.md index 9ada532fa..6dc4eba67 100644 --- a//sagemaker/latest/dg/smcluster-getting-started-slurm-console.md +++ b//sagemaker/latest/dg/smcluster-getting-started-slurm-console.md @@ -12,0 +13,4 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust +###### Note + +HyperPod now supports creating Slurm clusters without lifecycle scripts. You can create a fully functional cluster using AMI-based configuration, extend it with an extension script, or continue using custom lifecycle scripts for full control. + @@ -87,0 +92,4 @@ This section lists all the default settings for your cluster creation, including +###### Note + +Quick setup uses default lifecycle scripts automatically. The new AMI-based configuration option (no lifecycle scripts) is available only through Custom setup. If you want to create a cluster without lifecycle scripts, choose Custom setup and choose **None** under **Lifecycle scripts**. + @@ -139 +147,16 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi - 8. Choose **Add instance group**. + 8. For **Slurm partition name** (Compute groups only), enter the Slurm partition name for this compute instance group. Partitions act as logical queues that organize how jobs are scheduled across different sets of nodes. + + 9. Choose **Add instance group**. + + + + +### Lifecycle configuration - optional + +Configure how nodes in your cluster are provisioned. Your choice affects Amazon S3 bucket requirements, internet access needs, and provisioning complexity. HyperPod supports three node lifecycle configuration options, each offering a different level of control over the provisioning process. + + 1. For **Lifecycle scripts** , choose one of the following options to control how nodes are provisioned in your cluster: + + * **None** — HyperPod configures nodes automatically using AMI-based configuration. Slurm daemons, Docker, Enroot, Pyxis, Slurm accounting with MariaDB, SSH key generation and propagation, log rotation, and home directory setup are all configured without any scripts or Amazon S3 bucket. All software is pre-packaged in the AMI, so no internet access is required during provisioning. This is the simplest path for new clusters. + + * **Use default lifecycle scripts** — Default lifecycle scripts are uploaded to the chosen Amazon S3 bucket and used to provision nodes. This option uses the scripts from the [Awsome Distributed Training repository](https://github.com/awslabs/awsome-distributed-training/tree/main/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config). @@ -140,0 +164 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi + * **Use custom lifecycle scripts** — Choose lifecycle scripts from an Amazon S3 bucket. This corresponds to the `OnCreate` path in the API, where your scripts own the entire provisioning sequence, including when Slurm starts. HyperPod does not run AMI-based configuration when this option is selected. @@ -141,0 +166 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi +The following table summarizes the three options: @@ -142,0 +168,5 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi +Option | What HyperPod does | Amazon S3 bucket needed? | Internet access needed? +---|---|---|--- +**None** (AMI-based configuration) | Configures nodes automatically with Slurm and essential packages | No | No +**Use default lifecycle scripts** | Uploads and runs ADTR scripts from Amazon S3 | Yes | Yes +**Use custom lifecycle scripts** | Runs your scripts from Amazon S3; you own the full provisioning sequence | Yes | Depends on your scripts @@ -144 +174 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi -### Lifecycle scripts + 2. For **Extension script file in S3 -_optional_** (appears when you choose **None** under **Lifecycle scripts**), enter the Amazon S3 URI of your extension script. The extension script allows you to provision additional optional capabilities, such as observability, System Security Services Daemon (SSSD), and Amazon S3 bucket mounting, on top of default configurations without managing the entire set of lifecycle scripts. @@ -146 +176 @@ Ensure that you choose an instance type with sufficient quotas and enough unassi -You can choose to use the default lifecycle scripts or the custom lifecycle scripts, which will be stored in your Amazon S3 bucket. You can view the default lifecycle scripts in the [Awesome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/tree/main/1.architectures/7.sagemaker-hyperpod-eks/LifecycleScripts). To learn more about the lifecycle scripts, see [Customizing SageMaker HyperPod clusters using lifecycle scripts](./sagemaker-hyperpod-lifecycle-best-practices-slurm.html). +Enter the full Amazon S3 URI to the entry point script, for example: @@ -148 +178 @@ You can choose to use the default lifecycle scripts or the custom lifecycle scri - 1. For **Lifecycle scripts** , choose to use default or custom lifecycle scripts. + s3://DOC-EXAMPLE-BUCKET/extensions/run_extensions.sh @@ -150 +180 @@ You can choose to use the default lifecycle scripts or the custom lifecycle scri - 2. For **S3 bucket for lifecycle scripts** , choose to create a new bucket or use an existing bucket to store the lifecycle scripts. +HyperPod downloads the entire folder where the entry point script resides. Structure your Amazon S3 folder so that all supporting files are in the same directory as the entry point script. @@ -151,0 +182 @@ You can choose to use the default lifecycle scripts or the custom lifecycle scri +###### Note @@ -152,0 +184 @@ You can choose to use the default lifecycle scripts or the custom lifecycle scri +In the API, this corresponds to specifying `OnInitComplete` in `LifeCycleConfig` with `SourceS3Uri`. The console combines these into a single Amazon S3 URI field pointing directly to the entry point script. @@ -153,0 +186,16 @@ You can choose to use the default lifecycle scripts or the custom lifecycle scri +###### Tip + +For ready-to-use extension scripts, see the [Extensions folder](https://github.com/awslabs/awsome-distributed-training/tree/main/1.architectures/5.sagemaker-hyperpod/Extensions) in the Awsome Distributed Training repository. The `run_extensions.sh` script orchestrates multiple capabilities with simple boolean toggles to enable or disable each one. + + 3. For **S3 bucket for lifecycle scripts** (appears when you choose **Use default lifecycle scripts** or **Use custom lifecycle scripts**), choose to create a new bucket or use an existing bucket to store the lifecycle scripts. + + + + +###### Note + +Optional node lifecycle configuration is supported only for Slurm-orchestrated clusters. Amazon EKS-orchestrated clusters and Slurm clusters using Continuous `NodeProvisioningMode` continue to require lifecycle scripts on every instance group. + +###### Note + +The **None** option with an extension script and the **Use custom lifecycle scripts** option are mutually exclusive. You cannot combine AMI-based configuration with extension script and custom lifecycle scripts on the same instance group. In the API, this means `OnCreate` and `OnInitComplete` cannot be specified together. @@ -161 +209 @@ Choose or create an IAM role that allows HyperPod to run and access necessary AW -Configure the FSx for Lustre file system to be provisioned on the HyperPod cluster. +Configure the FSx for Lustre file system to be provisioned on the HyperPod cluster. FSx configuration is optional for cluster creation but recommended for production ML workloads. @@ -175,0 +224,4 @@ Configure the FSx for Lustre file system to be provisioned on the HyperPod clust +###### Note + +When using AMI-based configuration (choosing **None** under **Lifecycle scripts**) or an extension script, HyperPod handles FSx for Lustre mounting automatically. When using custom lifecycle scripts, your scripts are responsible for mounting the filesystem. + @@ -197 +249,5 @@ However, if you have created a cluster with reserved compute capacity, the statu -To clean up the lifecycle scripts from the S3 bucket used for this tutorial, go to the S3 bucket you used during cluster creation and remove the files entirely. +If you used **Use default lifecycle scripts** or **Use custom lifecycle scripts** , go to the Amazon S3 bucket you used during cluster creation and remove the lifecycle script files. + +If you used **None** (AMI-based configuration only) without an extension script, no Amazon S3 cleanup is needed for lifecycle scripts. + +If you used **None** with an extension script, clean up the extension script files from the Amazon S3 bucket you specified.