AWS sagemaker documentation change
Summary
Updated SageMaker HyperPod Slurm cluster creation documentation to introduce API-driven configuration. Removed requirement for separate provisioning_parameters.json file. Added SlurmConfig fields, FSx storage options, VPC configuration details, validation error handling, and comprehensive API response examples.
Security assessment
The changes focus on configuration methodology improvements and operational enhancements. No evidence of security vulnerability fixes or incident response. The VPC/SecurityGroup configuration mentions are standard network isolation practices, not new security features. Drift detection in SlurmConfigStrategy improves configuration integrity but isn't presented as a security control.
Diff
diff --git a/sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md b/sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md index 964f6dba7..3a2318ba4 100644 --- a//sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md +++ b//sagemaker/latest/dg/smcluster-getting-started-slurm-cli.md @@ -5 +5 @@ -Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and clean resources +Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and clean resourcesRelated topics @@ -13 +13,3 @@ Create your first SageMaker HyperPod cluster using the AWS CLI commands for Hype -The following tutorial demonstrates how to create a new SageMaker HyperPod cluster and set it up with Slurm through the [AWS CLI commands for SageMaker HyperPod](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-cli). Following the tutorial, you'll create a HyperPod cluster with three Slurm nodes, `my-controller-group`, `my-login-group`, and `worker-group-1`. +The following tutorial demonstrates how to create a new SageMaker HyperPod cluster and set it up with Slurm through the [AWS CLI commands for SageMaker HyperPod](./sagemaker-hyperpod-ref.html#sagemaker-hyperpod-ref-cli). Following the tutorial, you'll create a HyperPod cluster with three Slurm nodes: `my-controller-group`, `my-login-group`, and `worker-group-1`. + +With the API-driven configuration approach, you define Slurm node types and partition assignments directly in the CreateCluster API request using `SlurmConfig`. This eliminates the need for a separate `provisioning_parameters.json` file and provides built-in validation, drift detection, and per-instance-group FSx configuration. @@ -25 +27 @@ The S3 bucket path should start with a prefix `sagemaker-`, because the [IAM rol -If you are starting from scratch, use sample lifecycle scripts provided in the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/). The following sub-steps show how to download, what to modify, and how to upload the sample lifecycle scripts to an Amazon S3 bucket. +If you are starting from scratch, use sample lifecycle scripts provided in the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/). The following sub-steps show how to download and upload the sample lifecycle scripts to an Amazon S3 bucket. @@ -37,16 +39 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - 3. Write a Slurm configuration file and save it as `provisioning_parameters.json`. In the file, specify basic Slurm configuration parameters to properly assign Slurm nodes to the SageMaker HyperPod cluster instance groups. In this tutorial, set up three Slurm nodes named `my-controller-group`, `my-login-group`, and `worker-group-1`, as shown in the following example configuration `provisioning_parameters.json`. - - { - "version": "1.0.0", - "workload_manager": "slurm", - "controller_group": "my-controller-group", - "login_group": "my-login-group", - "worker_groups": [ - { - "instance_group_name": "worker-group-1", - "partition_name": "partition-1" - } - ] - } - - 4. Upload the scripts to `s3://sagemaker-`<unique-s3-bucket-name>`/`<lifecycle-script-directory>`/src`. You can do so by using the Amazon S3 console, or by running the following AWS CLI Amazon S3 command. + 3. Upload the scripts to `s3://sagemaker-`<unique-s3-bucket-name>`/`<lifecycle-script-directory>`/src`. You can do so by using the Amazon S3 console, or by running the following AWS CLI Amazon S3 command. @@ -58 +45,9 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - 2. Prepare a [CreateCluster](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCluster.html) request file in JSON format and save as `create_cluster.json`. The following request template aligns with the Slurm node configuration defined in the `provisioning_parameters.json` in Step 1.c. For `ExecutionRole`, provide the ARN of the IAM role you created with the managed `AmazonSageMakerClusterInstanceRolePolicy` in [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). +###### Note + +With API-driven configuration, you do not need to create or upload a `provisioning_parameters.json` file. The Slurm configuration is defined directly in the CreateCluster API request in the next step. + + 2. Prepare a [CreateCluster](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateCluster.html) request file in JSON format and save as `create_cluster.json`. + +With API-driven configuration, you specify the Slurm node type and partition assignment for each instance group using the `SlurmConfig` field. You also configure the cluster-level Slurm settings using `Orchestrator.Slurm`. + +For `ExecutionRole`, provide the ARN of the IAM role you created with the managed `AmazonSageMakerClusterInstanceRolePolicy` in [Prerequisites for using SageMaker HyperPod](./sagemaker-hyperpod-prerequisites.html). @@ -61 +55,0 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - // Required: Specify the name of the cluster. @@ -63 +56,0 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - // Required: Configure instance groups to be launched in the cluster @@ -66 +58,0 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - // Required: Specify the basic configurations to set up a controller node. @@ -69,0 +62,3 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp + "SlurmConfig": { + "NodeType": "Controller" + }, @@ -74,2 +69 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - "ExecutionRole": "${ROLE}", - // Optional: Configure an additional storage per instance group. + "ExecutionRole": "arn:aws:iam::<account-id>:role/HyperPodExecutionRole", @@ -78,2 +71,0 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - // Attach an additional EBS volume to each instance within the instance group. - // The default mount path for the additional EBS volume is /opt/sagemaker. @@ -81,2 +73 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - // Specify an integer between 1 and 16384 in gigabytes (GB). - "VolumeSizeInGB": <integer> + "VolumeSizeInGB": 500 @@ -90,0 +82,3 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp + "SlurmConfig": { + "NodeType": "Login" + }, @@ -95 +89 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - "ExecutionRole": "${ROLE}" + "ExecutionRole": "arn:aws:iam::<account-id>:role/HyperPodExecutionRole" @@ -100,0 +95,4 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp + "SlurmConfig": { + "NodeType": "Compute", + "PartitionNames": ["partition-1"] + }, @@ -105 +103,69 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp - "ExecutionRole": "${ROLE}" + "ExecutionRole": "arn:aws:iam::<account-id>:role/HyperPodExecutionRole" + } + ], + "Orchestrator": { + "Slurm": { + "SlurmConfigStrategy": "Managed" + } + } + } + +**SlurmConfig fields:** + +Field | Description +---|--- +`NodeType` | The Slurm role for the instance group. Valid values: `Controller`, `Login`, `Compute` +`PartitionNames` | The Slurm partition(s) to assign compute nodes to. Only valid for `Compute` node type. + +**Orchestrator.Slurm fields:** + +Field | Description +---|--- +`SlurmConfigStrategy` | Controls how HyperPod manages `slurm.conf`. Valid values: `Managed` (default), `Overwrite`, `Merge` + +**SlurmConfigStrategy options:** + + * `Managed` (recommended): HyperPod fully manages `slurm.conf` and detects unauthorized changes (drift detection). Updates fail if drift is detected. + + * `Overwrite`: HyperPod overwrites `slurm.conf` on updates, ignoring any manual changes. + + * `Merge`: HyperPod preserves manual changes and merges them with API configuration. + +**Adding FSx for Lustre (optional):** + +To mount an FSx for Lustre filesystem to your compute nodes, add `FsxLustreConfig` to the `InstanceStorageConfigs` for the instance group. This requires a Custom VPC configuration. + + { + "InstanceGroupName": "worker-group-1", + "InstanceType": "ml.trn1.32xlarge", + "InstanceCount": 1, + "SlurmConfig": { + "NodeType": "Compute", + "PartitionNames": ["partition-1"] + }, + "InstanceStorageConfigs": [ + { + "FsxLustreConfig": { + "DnsName": "fs-0abc123def456789.fsx.us-west-2.amazonaws.com", + "MountPath": "/fsx", + "MountName": "abcdefgh" + } + } + ], + "LifeCycleConfig": { + "SourceS3Uri": "s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src", + "OnCreate": "on_create.sh" + }, + "ExecutionRole": "arn:aws:iam::<account-id>:role/HyperPodExecutionRole" + } + +**Adding FSx for OpenZFS (optional):** + +You can also mount FSx for OpenZFS filesystems: + + "InstanceStorageConfigs": [ + { + "FsxOpenZfsConfig": { + "DnsName": "fs-0xyz789abc123456.fsx.us-west-2.amazonaws.com", + "MountPath": "/shared" + } @@ -107,0 +174,35 @@ To learn more about the lifecycle script samples, see [Customizing SageMaker Hyp + +###### Note + +Each instance group can have at most one FSx for Lustre and one FSx for OpenZFS configuration. Different instance groups can mount different filesystems. + +**Adding VPC configuration (required for FSx):** + +If using FSx, you must specify a Custom VPC configuration: + + { + "ClusterName": "my-hyperpod-cluster", + "InstanceGroups": [ + { + "InstanceGroupName": "my-controller-group", + "InstanceType": "ml.c5.xlarge", + "InstanceCount": 1, + "SlurmConfig": { + "NodeType": "Controller" + }, + "LifeCycleConfig": { + "SourceS3Uri": "s3://sagemaker-<unique-s3-bucket-name>/<lifecycle-script-directory>/src", + "OnCreate": "on_create.sh" + }, + "ExecutionRole": "arn:aws:iam::<account-id>:role/HyperPodExecutionRole" + }, + ], + "Orchestrator": { + "Slurm": { + "SlurmConfigStrategy": "Managed" + } + }, + "VpcConfig": { + "SecurityGroupIds": ["sg-0abc123def456789a"], + "Subnets": ["subnet-0abc123def456789a"] + } @@ -115,0 +217,4 @@ This should return the ARN of the created cluster. + { + "ClusterArn": "arn:aws:sagemaker:us-west-2:111122223333:cluster/my-hyperpod-cluster" + } + @@ -117,0 +223,8 @@ If you receive an error due to resource limits, ensure that you change the insta +**Common validation errors:** + +Error | Resolution +---|--- +"Cluster must have exactly one InstanceGroup with Controller node type" | Ensure exactly one instance group has `SlurmConfig.NodeType`: `"Controller"` +"Partitions can only be assigned to Compute node types" | Remove `PartitionNames` from `Controller` or `Login` instance groups +"FSx configurations are only supported for Custom VPC" | Add `VpcConfig` to your request when using FSx + @@ -122 +235,63 @@ If you receive an error due to resource limits, ensure that you change the insta -After the status of the cluster turns to `InService`, proceed to the next step.