AWS sagemaker documentation change
Summary
Comprehensive rewrite of SageMaker HyperPod Slurm cluster documentation, adding detailed configuration concepts, node lifecycle options (AMI-based/Extension/Custom), FSx storage setup, step-by-step cluster creation tutorials, and validation error resolutions.
Security assessment
The changes document security features including IAM role enforcement ('sagemaker-' S3 bucket prefix requirement), SSH key generation/propagation for secure node access, and Slurm configuration strategies ('Managed' mode) that enforce API-defined security settings. However, no specific vulnerability fix or security incident response is mentioned.
Diff
diff --git a/sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md b/sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md index 02030b5be..9600c0565 100644 --- a//sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md +++ b//sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md @@ -7 +7 @@ -Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and clean resourcesRelated topics +Key conceptsCreate your clusterConnect to your clusterDelete the cluster and clean resourcesRelated topics @@ -11 +11 @@ Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and cl -Create your first SageMaker HyperPod cluster using the AWS CLI commands for HyperPod. +The following tutorial demonstrates how to create a new SageMaker HyperPod cluster with Slurm through the [AWS CLI commands for SageMaker HyperPod](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-cli). By the end of this tutorial, you will have a working Slurm cluster with a controller node, a login node, and a compute worker group, ready to schedule and run ML workloads. The tutorial covers Slurm topology setup, node lifecycle configuration options, optional FSx shared storage, and how to connect to your cluster. @@ -13 +13 @@ Create your first SageMaker HyperPod cluster using the AWS CLI commands for Hype -## Create your first SageMaker HyperPod cluster with Slurm +Before starting, ensure you have completed the [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html) (VPC, quotas, FSx) and [AWS Identity and Access Management for SageMaker HyperPod](./sagemaker-hyperpod-prerequisites-iam.html) (IAM roles, execution role with `AmazonSageMakerClusterInstanceRolePolicy`). @@ -15 +15 @@ Create your first SageMaker HyperPod cluster using the AWS CLI commands for Hype -The following tutorial demonstrates how to create a new SageMaker HyperPod cluster and set it up with Slurm through the [AWS CLI commands for SageMaker HyperPod](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-cli). Following the tutorial, you'll create a HyperPod cluster with three Slurm nodes: `my-controller-group`, `my-login-group`, and `worker-group-1`. +## Key concepts @@ -17 +17 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust -With the API-driven configuration approach, you define Slurm node types and partition assignments directly in the CreateCluster API request using `SlurmConfig`. This eliminates the need for a separate `provisioning_parameters.json` file and provides built-in validation, drift detection, and per-instance-group FSx configuration. +This section covers the core configuration concepts for creating a SageMaker HyperPod Slurm cluster. Understanding these concepts will help you make informed choices when configuring your cluster, but if you want to get started immediately you can jump directly to Create your cluster and refer back here as needed. @@ -19 +19 @@ With the API-driven configuration approach, you define Slurm node types and part - 1. First, prepare and upload lifecycle scripts to an Amazon S3 bucket. During cluster creation, HyperPod runs them in each instance group. Upload lifecycle scripts to Amazon S3 using the following command. +When creating a Slurm-orchestrated cluster, you make two independent configuration choices: @@ -21,3 +21,56 @@ With the API-driven configuration approach, you define Slurm node types and part - aws s3 sync \ - ~/local-dir-to-lifecycle-scripts/* \ - s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src + 1. **Slurm topology configuration** – How is the Slurm cluster topology (node roles, partitions) defined? + + 2. **Node lifecycle configuration** – How are nodes provisioned and customized? + + + + +For Slurm topology, this tutorial uses the API-driven configuration approach, where you define node roles and partitions directly in the `CreateCluster` request using `SlurmConfig` on each instance group and `Orchestrator.Slurm` at the cluster level. This is the recommended approach for new clusters. It provides a single source of truth, built-in validation, and partition configuration drift detection with no additional files to manage. Alternatively, you can use a legacy `provisioning_parameters.json` file stored in Amazon S3 for backward compatibility with existing clusters. For details on the legacy approach, see [SageMaker HyperPod Slurm configuration](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-slurm-configuration). + +For node lifecycle configuration, SageMaker HyperPod supports three options. In the simplest case you omit `LifeCycleConfig` entirely and HyperPod automatically configures nodes using AMI-based configuration, setting up Slurm and essential packages such as Docker, Enroot, and Pyxis for running ML workloads, with no scripts or Amazon S3 bucket required. If you need customizations on top of the AMI-based configuration, you can provide an extension script via `OnInitComplete` that runs after the configuration completes. For full control over the entire provisioning sequence, the `OnCreate` path lets your scripts own everything, including when Slurm starts. + +For ML workloads you will typically also need a shared high-performance filesystem for training data, checkpoints, and shared libraries. SageMaker HyperPod supports Amazon FSx for Lustre and FSx for OpenZFS, configured per instance group via `InstanceStorageConfigs`. FSx configuration is optional for cluster creation but recommended for production workloads. + +### Configuring Slurm topology via the API + +All examples in this tutorial use API-driven Slurm topology configuration, where you define the Slurm cluster structure directly in the `CreateCluster` API request rather than through a separate configuration file. + +A Slurm cluster requires at least a controller node (which runs the `slurmctld` daemon and coordinates job scheduling) and one or more compute nodes (which execute jobs). Optionally, you can add a login node to provide users with a dedicated access point for submitting and managing jobs without logging into the controller directly. In the API request, you assign each instance group its Slurm role using `SlurmConfig`, specifying whether the group serves as the controller, login, or compute node. Compute groups are also mapped to one or more Slurm partitions, which act as logical queues that organize how jobs are scheduled across different sets of nodes. + +At the cluster level, `Orchestrator.Slurm` controls how HyperPod manages the partition configuration in `slurm.conf`. You choose a strategy that determines whether HyperPod is the single source of truth for partition topology, whether it overwrites manual changes, or whether it merges API-defined configuration with any manual edits you've made. Here is a reference for the fields used. + +**SlurmConfig** (per instance group): + + + "SlurmConfig": { + "NodeType": "Controller | Login | Compute", + "PartitionNames": ["partition-name"] + } + +Field | Description +---|--- +`NodeType` | Required. The Slurm role for this instance group. Valid values: `Controller`, `Login`, `Compute`. Exactly one instance group must be `Controller`. +`PartitionNames` | Conditional. Slurm partition names. Required for `Compute` node type; not allowed for `Controller` or `Login`. + +**Orchestrator.Slurm** (cluster level): + + + "Orchestrator": { + "Slurm": { + "SlurmConfigStrategy": "Managed | Overwrite | Merge" + } + } + +`SlurmConfigStrategy` determines how HyperPod manages the partition-to-node mappings in `slurm.conf` on the controller node. When you create or update a cluster, HyperPod writes the partition configuration to `slurm.conf` based on the `SlurmConfig` you defined on each instance group, mapping compute instance groups to their assigned partitions and registering controller and login nodes with the appropriate Slurm roles. + +The strategy you choose controls what happens when the partition configuration in `slurm.conf` has been modified outside of the API, for example, by an administrator editing the file directly on the controller node. With `Managed`, HyperPod treats the API as the single source of truth and will detect and block updates if `slurm.conf` on disk has drifted. With `Overwrite`, HyperPod forces the API-defined configuration onto the controller, discarding any manual edits to `slurm.conf`. This is useful for recovering from an unintended change. With `Merge`, HyperPod preserves manual edits to `slurm.conf` and merges them with the API configuration, giving advanced users the flexibility to maintain custom `slurm.conf` settings alongside API-managed partitions. + +Strategy | Partition drift detection | Manual changes | Use case +---|---|---|--- +`Managed` (default) | Enabled; blocks updates if drift found | Not supported | Single source of truth +`Overwrite` | Disabled | Overwritten on update | Recovery from drift +`Merge` | Disabled | Preserved and merged | Custom `slurm.conf` needs + +###### Important + +Drift detection applies only to Slurm partition configuration in `slurm.conf` (partition-to-node mappings defined through the API). Changes to other `slurm.conf` settings, such as scheduling parameters, resource limits, or accounting configuration, are not monitored and will not be detected or reported by HyperPod. @@ -27 +80 @@ With the API-driven configuration approach, you define Slurm node types and part -The S3 bucket path should start with a prefix `sagemaker-`, because the [IAM role for SageMaker HyperPod](./sagemaker-hyperpod-prerequisites-iam.html#sagemaker-hyperpod-prerequisites-iam-role-for-hyperpod) with `AmazonSageMakerClusterInstanceRolePolicy` only allows access to Amazon S3 buckets that starts with the specific prefix. +If you prefer to define Slurm topology using a `provisioning_parameters.json` file instead of the API, omit `SlurmConfig` from instance groups and `Orchestrator.Slurm` from the cluster request, and upload the file to Amazon S3 alongside your node lifecycle scripts. For details, see [SageMaker HyperPod Slurm configuration](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-slurm-configuration). @@ -29 +82 @@ The S3 bucket path should start with a prefix `sagemaker-`, because the [IAM rol -If you are starting from scratch, use sample lifecycle scripts provided in the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/). The following sub-steps show how to download and upload the sample lifecycle scripts to an Amazon S3 bucket. +### Node lifecycle configuration options @@ -31 +84 @@ If you are starting from scratch, use sample lifecycle scripts provided in the [ - 1. Download a copy of the lifecycle script samples to a directory on your local computer. +When creating a SageMaker HyperPod Slurm cluster, you choose how each instance group's nodes are provisioned by configuring the `LifeCycleConfig` block in the `CreateCluster` request. SageMaker HyperPod supports three node lifecycle configuration options, each offering a different level of control over the provisioning process. @@ -33 +86 @@ If you are starting from scratch, use sample lifecycle scripts provided in the [ - git clone https://github.com/aws-samples/awsome-distributed-training/ +With **AMI-based configuration** only, you omit `LifeCycleConfig` entirely. HyperPod automatically configures nodes using AMI-based configuration, setting up Slurm, installing essential packages, and starting all required services. This is the simplest path and requires no Amazon S3 bucket or scripts. @@ -35 +88 @@ If you are starting from scratch, use sample lifecycle scripts provided in the [ - 2. Go into the directory [`1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config`](https://github.com/aws-samples/awsome-distributed-training/tree/main/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config), where you can find a set of lifecycle scripts. +With the **Extension** option, you specify `OnInitComplete` in `LifeCycleConfig` along with a `SourceS3Uri` pointing to your extension script in Amazon S3. HyperPod runs the full AMI-based configuration first, then executes your script. This lets you add customizations, such as monitoring agents, LDAP integration, or additional storage mounts, without managing the baseline provisioning. @@ -37 +90,15 @@ If you are starting from scratch, use sample lifecycle scripts provided in the [ - cd awsome-distributed-training/1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config +With the **Custom** option, you specify `OnCreate` in `LifeCycleConfig` along with a `SourceS3Uri` pointing to your full lifecycle script set in Amazon S3. HyperPod does not run AMI-based configuration and does not start Slurm. Your scripts own the entire provisioning sequence. This gives you complete control over what software is installed, how it is configured, and when Slurm starts. + +Node lifecycle option | Amazon S3 bucket needed? | Scripts to upload? | LifeCycleConfig in API? +---|---|---|--- +**AMI-based configuration** only (simplest) | No | No | Omit entirely +**Extension** (`OnInitComplete`) | Yes | Only your extension script | `OnInitComplete` \+ `SourceS3Uri` +**Custom** (`OnCreate`) | Yes | Full lifecycle script set | `OnCreate` \+ `SourceS3Uri` + +###### Note + +Optional node lifecycle configuration is supported only for Slurm-orchestrated clusters. EKS-orchestrated clusters and Slurm clusters using Continuous `NodeProvisioningMode` continue to require `LifeCycleConfig` with `OnCreate` and `SourceS3Uri` on every instance group. + +###### Note + +`OnCreate` and `OnInitComplete` are mutually exclusive. Specifying both on the same instance group results in a validation error. @@ -39 +106 @@ If you are starting from scratch, use sample lifecycle scripts provided in the [ -To learn more about the lifecycle script samples, see [Customizing SageMaker HyperPod clusters using lifecycle scripts](./sagemaker-hyperpod-lifecycle-best-practices-slurm.html). +### FSx and VPC configuration @@ -41 +108 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - 3. Upload the scripts to `s3://sagemaker-`<unique-s3-bucket-name>`/`<lifecycle-script-directory>`/src`. You can do so by using the Amazon S3 console, or by running the following AWS CLI Amazon S3 command. +For ML workloads, a shared high-performance filesystem is essential for storing training data, model checkpoints, and shared libraries across cluster nodes. SageMaker HyperPod supports Amazon FSx for Lustre and FSx for OpenZFS, configured per instance group via `InstanceStorageConfigs`. FSx filesystems reside in your VPC, so a custom VPC configuration (`VpcConfig`) is required when using FSx. @@ -43,3 +110,37 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - aws s3 sync \ - ~/local-dir-to-lifecycle-scripts/* \ - s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src +FSx configuration works with all three node lifecycle configuration options. When using AMI-based configuration or `OnInitComplete`, HyperPod handles FSx mounting automatically. When using `OnCreate`, your lifecycle scripts are responsible for mounting. + +**FSx for Lustre:** + + + "InstanceStorageConfigs": [ + { + "FsxLustreConfig": { + "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com", + "MountPath": "/fsx", + "MountName": "abcdefgh" + } + } + ] + +Field | Description +---|--- +`DnsName` | Required. The DNS name of the FSx for Lustre filesystem. +`MountPath` | Optional. The local mount path on the instance. Default: `/fsx` +`MountName` | Required. The mount name of the FSx for Lustre filesystem. Find this in the FSx for Lustre console or via `aws fsx describe-file-systems`. + +**FSx for OpenZFS:** + + + "InstanceStorageConfigs": [ + { + "FsxOpenZfsConfig": { + "DnsName": "fs-0xyz789abc123456.fsx.us-west-2.amazonaws.com", + "MountPath": "/shared" + } + } + ] + +Field | Description +---|--- +`DnsName` | Required. The DNS name of the FSx for OpenZFS filesystem. +`MountPath` | Optional. The local mount path on the instance. Default: `/home` @@ -49 +150,13 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp -With API-driven configuration, you do not need to create or upload a `provisioning_parameters.json` file. The Slurm configuration is defined directly in the CreateCluster API request in the next step. +Each instance group can have at most one FSx for Lustre and one FSx for OpenZFS configuration. Different instance groups can mount different filesystems. + +**VPC configuration** (required for FSx): + +Add `VpcConfig` at the cluster level in your `CreateCluster` request: + + + "VpcConfig": { + "SecurityGroupIds": ["sg-0abc123def456789a"], + "Subnets": ["subnet-0abc123def456789a"] + } + +For more information on setting up a VPC, see [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). For more information on FSx setup, see [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). @@ -51 +164 @@ With API-driven configuration, you do not need to create or upload a `provisioni - 2. Prepare a [CreateCluster](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCluster.html) request file in JSON format and save as `create_cluster.json`. +## Create your cluster @@ -53 +166 @@ With API-driven configuration, you do not need to create or upload a `provisioni -With API-driven configuration, you specify the Slurm node type and partition assignment for each instance group using the `SlurmConfig` field. You also configure the cluster-level Slurm settings using `Orchestrator.Slurm`. +This section walks you through creating a cluster using each of the three node lifecycle configuration options described in Node lifecycle configuration options. For most users, we recommend starting with **Option A** , AMI-based configuration only. It requires no scripts or Amazon S3 bucket and delivers a fully functional cluster out of the box. Choose Option B if you need to add customizations on top of the AMI-based configuration, or Option C if you need full control over the provisioning process. @@ -55 +168,22 @@ With API-driven configuration, you specify the Slurm node type and partition ass -For `ExecutionRole`, provide the ARN of the IAM role you created with the managed `AmazonSageMakerClusterInstanceRolePolicy` in [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). +For `ExecutionRole` in all examples, provide the ARN of the IAM role you created with the managed `AmazonSageMakerClusterInstanceRolePolicy` in [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). + +### Option A: AMI-based configuration only (without lifecycle configuration) + +This is the simplest path. No Amazon S3 bucket, scripts, or configuration files are needed. SageMaker HyperPod configures nodes automatically using AMI-based configuration, installing essential software and applying configurations so the cluster is ready to run ML workloads out of the box. All software packages are embedded in the AMI, so no internet access is required during provisioning. + +The following table lists the capabilities included in AMI-based configuration: + +Capability | Description +---|--- +Slurm daemons| Controller and compute daemons started automatically +Docker| Container runtime for building and running ML containers +Enroot| Rootless container execution for Slurm workloads +Pyxis| Slurm plugin for container integration +Slurm accounting| Configures Slurm job accounting for tracking job history and resource consumption +MariaDB| Deploys MariaDB on the controller node as the backing database for Slurm accounting