AWS sagemaker documentation change
Summary
Updated documentation for SageMaker HyperPod topology-aware scheduling, changing from manual UltraServer/block plugin configuration to automatic topology selection based on instance types. Added details about topology/tree plugin, block plugin, mixed instance type handling, dynamic updates, and manual override options.
Security assessment
This change is a documentation update about performance optimization and cluster management features for SageMaker HyperPod. It describes automatic topology plugin selection, configuration details for tree and block plugins, and guidance for mixed instance type clusters. There is no mention of security vulnerabilities, patches, or security incidents. The changes focus on performance tuning, scheduling optimization, and cluster management best practices without any security context.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-topology.md b/sagemaker/latest/dg/sagemaker-hyperpod-topology.md index 782f6f120..b028921f6 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-topology.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-topology.md @@ -51 +51 @@ Using these labels, you can use topology-aware scheduling in HyperPod task gover -Slurm provides built-in plugins for network topology awareness. UltraServer architecture in SageMaker HyperPod supports the block plugin. +Slurm provides built-in plugins for network topology awareness. SageMaker HyperPod automatically selects and configures the appropriate topology plugin based on the instance types in your cluster. @@ -53 +53 @@ Slurm provides built-in plugins for network topology awareness. UltraServer arch -### Using the topology/block Plugin +### Automatic topology selection @@ -55 +55,41 @@ Slurm provides built-in plugins for network topology awareness. UltraServer arch -NVIDIA developed a topology/block plugin that provides hierarchical scheduling across blocks of nodes with the following characteristics: +When you create a HyperPod Slurm cluster, the system inspects all instance groups and their associated instance types, identifies the GPU communication characteristics of each instance type, and configures Slurm with the appropriate topology plugin. This process runs automatically and does not require any configuration. + +HyperPod manages topology through a dynamically generated `topology.conf` file. As the cluster evolves through scaling operations or node replacements, HyperPod continuously reconciles the topology configuration to reflect the current cluster state. For more information, see Dynamic topology updates. + +### Using the topology/tree plugin + +The `topology/tree` plugin models hierarchical communication structures with multiple bandwidth tiers. Tree topology enables Slurm to place jobs in a way that minimizes cross-tier communication and maximizes locality. + +Tree topology is used for instance types with hierarchical interconnects, where distributed training workloads benefit from locality-aware placement. This includes instance types such as `ml.p5.48xlarge`, `ml.p5e.48xlarge`, and `ml.p5en.48xlarge`. + +SageMaker HyperPod automatically configures the `topology/tree` plugin when your cluster uses these instance types. The generated `topology.conf` maps nodes into a switch hierarchy that reflects the communication tiers of your hardware. + +Ensure your `slurm.conf` includes: + + + TopologyPlugin=topology/tree + +#### Configuration + +SageMaker HyperPod automatically configures the `topology/tree` plugin based on information provided by Amazon EC2. For more details about Amazon EC2 topology, see [Amazon EC2 instance topology](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-topology.html). + +When the `topology/tree` plugin is used, the Slurm `topology.conf` looks like the following: + + + SwitchName=nn-6fe9d8a965d34d181 Switches=nn-0b53107754517bf0e + + SwitchName=nn-0b53107754517bf0e Switches=nn-424c855d4ad825aa4,nn-95acd7c656329fc30 + + SwitchName=nn-424c855d4ad825aa4 Nodes=ip-10-1-111-198 + SwitchName=nn-95acd7c656329fc30 Nodes=ip-10-1-53-231 + +#### Usage + +When the `topology/tree` plugin is configured, Slurm tries to allocate machines that are close to each other. You can force Slurm to allocate machines on a single switch by passing the `--switch` command line parameter to `sbatch` or `srun`: + + + sbatch --switch=1 .... + +### Using the topology/block plugin + +NVIDIA developed a `topology/block` plugin that provides hierarchical scheduling across blocks of nodes with the following characteristics: @@ -71,0 +112,4 @@ This plugin allocates nodes based on the defined network topology. +Block topology models uniform, high-bandwidth communication domains where all GPUs participate in a single high-speed domain with near-uniform latency. Block topology treats all nodes as part of a single cohesive communication unit. UltraServer architecture in SageMaker HyperPod supports the block plugin. + +Block topology is used for instance types such as `ml.p6e-gb200.NVL72` and `ml.p6e-gb300.NVL72`. + @@ -74 +118 @@ This plugin allocates nodes based on the defined network topology. -To configure topology-aware scheduling with the topology/block plugin, +SageMaker HyperPod automatically configures the `topology/block` plugin. If you want to configure the plugin manually, specify the following in the `topology.conf` file in your Slurm configuration directory: @@ -76 +119,0 @@ To configure topology-aware scheduling with the topology/block plugin, - * SageMaker HyperPod automatically configures the topology/block plugin. If you want to configure the plugin, specify the following in the topology.conf file in your Slurm configuration directory: @@ -84,4 +125 @@ To configure topology-aware scheduling with the topology/block plugin, - * Ensure your `slurm.conf` includes: - - TopologyPlugin=topology/block - +Ensure your `slurm.conf` includes: @@ -89,0 +128 @@ To configure topology-aware scheduling with the topology/block plugin, + TopologyPlugin=topology/block @@ -122 +161,46 @@ The following are sample scenarios you might consider when thinking about alloca - sbatch -N24 —segment=12 + sbatch -N24 --segment=12 + +**With --exclusive=topo, job must be placed on block with no other jobs** + + + sbatch -N12 --exclusive=topo + +### Topology selection for clusters with mixed instance types + +HyperPod currently uses Slurm 24.11, which supports only a single topology configuration per cluster. This means that per-partition topology selection is not supported, multiple topology models cannot coexist within a single cluster, and all nodes must conform to a single topology definition. + +When your cluster contains multiple instance types, HyperPod selects a topology that is compatible across all of them. The following table shows an example of how HyperPod resolves topology for a cluster with mixed instance types. + +Instance group | Instance type | Preferred topology +---|---|--- +IG-1 | ml.p5.48xlarge | Tree +IG-2 | ml.p6e-gb300.NVL72 | Block + +In this example, block topology is optimal for ml.p6e-gb300.NVL72, but tree topology is compatible with both ml.p5.48xlarge and ml.p6e-gb300.NVL72. HyperPod selects tree topology as the cluster-wide topology to ensure that all nodes can participate in scheduling correctly and no instance type is excluded or misrepresented. + +###### Important + +For workloads where topology-aware scheduling is critical to performance, we recommend creating separate clusters for each instance type rather than combining different instance types in a single cluster. This ensures that each cluster uses the optimal topology for its hardware, delivering the best possible workload performance. For example, instead of combining ml.p5.48xlarge and ml.p6e-gb300.NVL72 instances in a single cluster where tree topology is selected as a compatible compromise, create a dedicated cluster for each instance type so that each uses its ideal topology model. + +### Disable or change topology plugin + +When a Slurm cluster is created, HyperPod automatically selects the optimal topology plugin. To manually change the topology plugin, update the `TopologyPlugin` value in `slurm.conf` on the controller node. + +Example: + + + # Set this value to disable topology plugin + TopologyPlugin=topology/flat + +### Dynamic topology updates + +Topology-aware scheduling continuously maintains topology correctness as your cluster changes. The topology is automatically recalculated and the `topology.conf` file is regenerated when any of the following events occur: + + * **Scale-up** : New nodes are added to the cluster. + + * **Scale-down** : Nodes are removed from the cluster. + + * **Node replacement** : Failed or unhealthy nodes are replaced, or nodes are manually replaced using the [BatchReplaceClusterNodes](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_BatchReplaceClusterNodes.html) API. + + + @@ -124 +208 @@ The following are sample scenarios you might consider when thinking about alloca -**With —exclusive=topo, job must be placed on block with no other jobs** +When the topology is updated, new nodes are incorporated into the correct topology structure, removed nodes are pruned, and the Slurm configuration is updated without requiring manual intervention. This ensures that the topology always reflects the actual cluster state. @@ -125,0 +210 @@ The following are sample scenarios you might consider when thinking about alloca +###### Note @@ -127 +212 @@ The following are sample scenarios you might consider when thinking about alloca - sbatch -N12 —exclusive=topo +Advanced users can override the topology behavior by logging into the Slurm controller node and manually modifying `slurm.conf` and `topology.conf`. However, manual changes may be overwritten by HyperPod during subsequent cluster updates, including scaling operations, node replacements, and other cluster lifecycle events. If you modify these files manually, verify your changes after any cluster update.