AWS sagemaker documentation change
Summary
Restructured FAQ into sections, added new Q&As covering EFA security groups setup, cgroups for OOM prevention, node state management, and resource cleanup scripts.
Security assessment
The changes document security-related configurations such as EFA security group setup (network security controls) and cgroups implementation (resource isolation). While these improve security posture, there is no evidence they address specific existing vulnerabilities. The EFA guidance helps prevent insecure network configurations, and cgroups documentation describes isolation features.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-faq-slurm.md b/sagemaker/latest/dg/sagemaker-hyperpod-faq-slurm.md index a76b3dd1f..6e7273456 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-faq-slurm.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-faq-slurm.md @@ -4,0 +5,2 @@ +Why can't I find log groups of my SageMaker HyperPod cluster in Amazon CloudWatch?What particular configurations does HyperPod manage in Slurm configuration files such as slurm.conf and gres.conf?How do I run Docker on Slurm nodes on HyperPod?Why does my parallel training job fail when I use NVIDIA Collective Communications Library (NCCL) with Slurm on SageMaker HyperPod platform?How do I use local NVMe store of P instances for launching Docker or Enroot containers with Slurm?How to set up EFA security groups?How do I monitor my HyperPod cluster nodes? Are there any CloudWatch metrics exported from HyperPod?Can I add an additional storage to the HyperPod cluster nodes? The cluster instances have limited local instance store.Why are my compute nodes showing as "DOWN" or "DRAINED" after a reboot?Why do my nodes keep getting drained due to Out of Memory (OOM) issues?How can I ensure resources are properly cleaned up after jobs complete? + @@ -9 +11,28 @@ Use the following frequently asked questions to troubleshoot problems with using -**Q. Why can I not find log groups of my SageMaker HyperPod cluster in Amazon CloudWatch?** +###### Amazon SageMaker HyperPod FAQs + + * Why can't I find log groups of my SageMaker HyperPod cluster in Amazon CloudWatch? + + * What particular configurations does HyperPod manage in Slurm configuration files such as slurm.conf and gres.conf? + + * How do I run Docker on Slurm nodes on HyperPod? + + * Why does my parallel training job fail when I use NVIDIA Collective Communications Library (NCCL) with Slurm on SageMaker HyperPod platform? + + * How do I use local NVMe store of P instances for launching Docker or Enroot containers with Slurm? + + * How to set up EFA security groups? + + * How do I monitor my HyperPod cluster nodes? Are there any CloudWatch metrics exported from HyperPod? + + * Can I add an additional storage to the HyperPod cluster nodes? The cluster instances have limited local instance store. + + * Why are my compute nodes showing as "DOWN" or "DRAINED" after a reboot? + + * Why do my nodes keep getting drained due to Out of Memory (OOM) issues? + + * How can I ensure resources are properly cleaned up after jobs complete? + + + + +## Why can't I find log groups of my SageMaker HyperPod cluster in Amazon CloudWatch? @@ -52 +81 @@ However, if you use custom paths for collecting logs from lifecycle provisioning -**Q. What particular configurations does HyperPod manage in Slurm configuration files such as`slurm.conf` and `gres.conf`?** +## What particular configurations does HyperPod manage in Slurm configuration files such as `slurm.conf` and `gres.conf`? @@ -72 +101 @@ Also, to enable the [Auto-resume](./sagemaker-hyperpod-resiliency-slurm.html#sag -**Q. How do I run Docker on Slurm nodes on HyperPod?** +## How do I run Docker on Slurm nodes on HyperPod? @@ -76 +105 @@ To help you run Docker on your Slurm nodes running on HyperPod, the HyperPod ser -**Q. Why does my parallel training job fail when I use NVIDIA Collective Communications Library (NCCL) on SageMaker HyperPod platform on the Slurm framework?** +## Why does my parallel training job fail when I use NVIDIA Collective Communications Library (NCCL) with Slurm on SageMaker HyperPod platform? @@ -161 +190 @@ For more information about the Epilog parameter, see [Slurm documentation](https -**Q. How do I use local NVMe store of P instances for launching Docker or Enroot containers with Slurm?** +## How do I use local NVMe store of P instances for launching Docker or Enroot containers with Slurm? @@ -165 +194 @@ Because the default root volume of your head node usually is limited by 100GB EB -**Q. How to set up EFA security groups?** +## How to set up EFA security groups? @@ -169 +198 @@ If you want to create a HyperPod cluster with EFA-enabled instances, make sure t -**Q. How do I monitor my HyperPod cluster nodes? Is there any CloudWatch metrics exported from HyperPod?** +## How do I monitor my HyperPod cluster nodes? Are there any CloudWatch metrics exported from HyperPod? @@ -173 +202 @@ To gain observability into the resource utilization of your HyperPod cluster, we -**Q. Can I add an additional storage to the HyperPod cluster nodes? The cluster instances have limited local instance store.** +## Can I add an additional storage to the HyperPod cluster nodes? The cluster instances have limited local instance store. @@ -176,0 +206,37 @@ If the default instance storage is insufficient for your workload, you can confi +## Why are my compute nodes showing as "DOWN" or "DRAINED" after a reboot? + +This typically occurs when nodes are rebooted using `sudo reboot` instead of Slurm's control interface. To properly reboot nodes, use the Slurm command `scontrol reboot nextstate=resume <list_of_nodes>`. This ensures Slurm maintains proper control of the node state and resumes normal operation after reboot. + +For GPU instances (like NVIDIA P5), this can also happen if the node can't complete its boot process within Slurm's default time limit (60 seconds). To resolve this, increase the `TimeToResume` parameter in `slurm.conf` to 300 seconds. This gives GPU instances sufficient time to boot and initialize drivers. + +## Why do my nodes keep getting drained due to Out of Memory (OOM) issues? + +OOM issues occur when jobs exceed the node's memory capacity. To prevent this, implement `cgroups` to enforce memory limits per job. This prevents a single job from affecting the entire node and improves isolation and stability. + +Example setup in `slurm.conf`: + + + TaskPlugin=task/cgroup + +Example setup in `cgroup.conf`: + + + CgroupAutomount=yes + ConstrainCores=yes + CgroupPlugin=autodetect + ConstrainDevices=yes + ConstrainRAMSpace=yes + ConstrainSwapSpace=yes + SignalChildrenProcesses=yes + MaxRAMPercent=99 + MaxSwapPercent=80 + MinRAMSpace=100 + +For more information, see [Control Group in Slurm](https://slurm.schedmd.com/cgroups.html), [Cgroup and PAM-based login control for Slurm compute nodes](https://github.com/aws-samples/awsome-distributed-training/blob/main/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config/utils/pam_adopt_cgroup_wheel.sh#L197), and [Configure Cgroups for Slurm](https://catalog.workshops.aws/sagemaker-hyperpod/en-US/07-tips-and-tricks/16-enable-cgroups). + +## How can I ensure resources are properly cleaned up after jobs complete? + +Implement epilogue scripts to automatically clean up resources after jobs complete. Resources might not be cleared correctly when jobs crash unexpectedly, contain bugs that prevent normal cleanup, or when shared memory buffers (include those shared between processes and GPU drivers) retain allocated. + +Epilogue scripts can perform tasks such as clearing GPU memory, removing temporary files, and unmounting file systems. These scripts have limitations when resources are not exclusively allocated to a single job. For detailed instructions and sample scripts, see the second bullet point of the question Why does my parallel training job fail when I use NVIDIA Collective Communications Library (NCCL) with Slurm on SageMaker HyperPod platform?. For more information, see [Enable Slurm epilog script](https://catalog.workshops.aws/sagemaker-hyperpod/en-US/07-tips-and-tricks/18-slurm-epilogue). +