AWS sagemaker documentation change
Summary
Replaced 'SageMaker HyperPod forms' section with comprehensive 'SageMaker HyperPod Slurm configuration' documentation. Added two configuration approaches (API-driven and legacy), detailed API parameters for Slurm node types/partitions, FSx storage configuration, strategy management, and complete implementation examples.
Security assessment
The changes focus on operational configuration improvements for Slurm workload management. While new features like drift detection could indirectly help maintain configuration integrity, there's no evidence of addressing specific vulnerabilities or security incidents. The documentation update doesn't mention security controls, vulnerability mitigations, or security incidents.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-ref.md b/sagemaker/latest/dg/sagemaker-hyperpod-ref.md index 7299d7f8f..b46107597 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-ref.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-ref.md @@ -5 +5 @@ -SageMaker HyperPod pricingSageMaker HyperPod APIsSageMaker HyperPod formsSageMaker HyperPod DLAMISageMaker HyperPod API permissions referenceSageMaker HyperPod commands in AWS CLISageMaker HyperPod Python modules in AWS SDK for Python (Boto3) +SageMaker HyperPod pricingSageMaker HyperPod APIsSageMaker HyperPod Slurm configurationSageMaker HyperPod DLAMISageMaker HyperPod API permissions referenceSageMaker HyperPod commands in AWS CLISageMaker HyperPod Python modules in AWS SDK for Python (Boto3) @@ -17 +17 @@ Find more information and references about using SageMaker HyperPod in the follo - * SageMaker HyperPod forms + * SageMaker HyperPod Slurm configuration @@ -69 +69 @@ The following list is a full set of SageMaker HyperPod APIs for submitting actio -## SageMaker HyperPod forms +## SageMaker HyperPod Slurm configuration @@ -71 +71 @@ The following list is a full set of SageMaker HyperPod APIs for submitting actio -To configure the Slurm workload manager tool on HyperPod, you should create a Slurm configuration file required by HyperPod using the provided form. +HyperPod supports two approaches for configuring Slurm on your cluster. Choose the approach that best fits your needs. @@ -73 +73,350 @@ To configure the Slurm workload manager tool on HyperPod, you should create a Sl -### Configuration form for provisioning Slurm nodes on HyperPod +**Approach** | **Description** | **Recommended For** +---|---|--- +API-driven configuration | Define Slurm configuration directly in the CreateCluster and UpdateCluster API requests | New clusters; simplified management +Legacy configuration | Use a separate `provisioning_parameters.json` file stored in Amazon S3 | Existing clusters; backward compatibility + +### API-driven Slurm configuration (Recommended) + +With API-driven configuration, you define Slurm node types, partition assignments, and filesystem mounts directly in the CreateCluster and UpdateCluster API requests. This approach provides: + + * **Single source of truth** – All configuration in the API request + + * **No S3 file management** – No need to create or maintain `provisioning_parameters.json` + + * **Built-in validation** – API validates Slurm topology before cluster creation + + * **Drift detection** – Detects unauthorized changes to `slurm.conf` + + * **Per-instance-group storage** – Configure different FSx filesystems for different instance groups + + * **FSx for OpenZFS support** – Mount OpenZFS filesystems in addition to FSx for Lustre + + + + +#### SlurmConfig (per instance group) + +Add `SlurmConfig` to each instance group to define the Slurm node type and partition assignment. + + + "SlurmConfig": { + "NodeType": "Controller | Login | Compute", + "PartitionNames": ["string"] + } + +**Parameters:** + + * `NodeType` – Required. The Slurm node type for this instance group. Valid values: + + * `Controller` – Slurm controller (head) node. Runs the `slurmctld` daemon. Exactly one instance group must have this node type. + + * `Login` – Login node for user access. Optional. At most one instance group can have this node type. + + * `Compute` – Worker nodes that execute jobs. Can have multiple instance groups with this node type. + +###### Important + +`NodeType` is immutable. Once set during cluster creation, it cannot be changed. To use a different node type, create a new instance group. + + * `PartitionNames` – Conditional. An array of Slurm partition names. Required for `Compute` node types; not allowed for `Controller` or `Login` node types. Currently supports a single partition name per instance group. + +###### Note + +All nodes are automatically added to the universal `dev` partition in addition to their specified partition. + + + + +**Example:** + + + { + "InstanceGroupName": "gpu-compute", + "InstanceType": "ml.p4d.24xlarge", + "InstanceCount": 8, + "SlurmConfig": { + "NodeType": "Compute", + "PartitionNames": ["gpu-training"] + }, + "LifeCycleConfig": { + "SourceS3Uri": "s3://sagemaker-bucket/lifecycle/src/", + "OnCreate": "on_create.sh" + }, + "ExecutionRole": "arn:aws:iam::111122223333:role/HyperPodRole" + } + +#### Orchestrator.Slurm (cluster level) + +Add `Orchestrator.Slurm` to the cluster configuration to specify how HyperPod manages the `slurm.conf` file. + + + "Orchestrator": { + "Slurm": { + "SlurmConfigStrategy": "Managed | Overwrite | Merge" + } + } + +**Parameters:** + + * `SlurmConfigStrategy` – Required when `Orchestrator.Slurm` is provided. Controls how HyperPod manages the `slurm.conf` file on the controller node. Valid values: + + * `Managed` (default) – HyperPod fully controls the partition-node mappings in `slurm.conf`. Drift detection is enabled: if the current `slurm.conf` differs from the expected configuration, UpdateCluster fails with an error. Use this strategy when you want HyperPod to be the single source of truth for Slurm configuration. + + * `Overwrite` – HyperPod forces the API configuration to be applied, overwriting any manual changes to `slurm.conf`. Drift detection is disabled. Use this strategy to recover from drift or reset the cluster to a known state. + + * `Merge` – HyperPod preserves manual `slurm.conf` changes and merges them with the API configuration. Drift detection is disabled. Use this strategy if you need to make manual Slurm configuration changes that should persist across updates. + + + + +###### Note + +If `Orchestrator.Slurm` is omitted from the request, the default behavior is `Managed` strategy. + +###### Tip + +You can change `SlurmConfigStrategy` at any time using UpdateCluster. There is no lock-in to a specific strategy. + +**Example:** + + + { + "ClusterName": "my-hyperpod-cluster", + "InstanceGroups": [...], + "Orchestrator": { + "Slurm": { + "SlurmConfigStrategy": "Managed" + } + } + } + +#### SlurmConfigStrategy comparison + +**Strategy** | **Drift Detection** | **Manual Changes** | **Use Case** +---|---|---|--- +`Managed` | Enabled – blocks updates if drift detected | Blocked | HyperPod managed +`Overwrite` | Disabled | Overwritten | Recovery from drift; reset to known state +`Merge` | Disabled | Preserved | Advanced users with custom `slurm.conf` needs + +#### FSx configuration via InstanceStorageConfigs + +With API-driven configuration, you can configure FSx filesystems per instance group using `InstanceStorageConfigs`. This allows different instance groups to mount different filesystems. + +**Prerequisites:** + + * Your cluster must use a custom VPC (via `VpcConfig`). FSx filesystems reside in your VPC, and the platform-managed VPC cannot reach them. + + * At least one instance group must have `SlurmConfig` with `NodeType: Controller`. + + + + +##### FsxLustreConfig + +Configure FSx for Lustre filesystem mounting for an instance group. + + + "InstanceStorageConfigs": [ + { + "FsxLustreConfig": { + "DnsName": "string", + "MountPath": "string", + "MountName": "string" + } + } + ] + +**Parameters:** + + * `DnsName` – Required. The DNS name of the FSx for Lustre filesystem. Example: `fs-0abc123def456789.fsx.us-west-2.amazonaws.com` + + * `MountPath` – Optional. The local mount path on the instance. Default: `/fsx` + + * `MountName` – Required. The mount name of the FSx for Lustre filesystem. You can find this in the Amazon FSx console or by running `aws fsx describe-file-systems`. + + + + +##### FsxOpenZfsConfig + +Configure FSx for OpenZFS filesystem mounting for an instance group. + + + "InstanceStorageConfigs": [ + { + "FsxOpenZfsConfig": { + "DnsName": "string", + "MountPath": "string" + } + } + ] + +**Parameters:**