AWS sagemaker documentation change
Summary
Restructured HyperPod cluster creation documentation with expanded setup options, added detailed configuration steps for Slurm orchestration, and clarified resource management procedures
Security assessment
Added documentation about security group configuration ("create security group or choose up to five security groups configured with rules") and IAM role requirements. While these are security-related configurations, there's no evidence of addressing a specific vulnerability or security incident. The changes improve security documentation but don't indicate remediation of a security issue.
Diff
diff --git a/sagemaker/latest/dg/smcluster-getting-started-slurm-console.md b/sagemaker/latest/dg/smcluster-getting-started-slurm-console.md index d1b5b6d35..62e539441 100644 --- a//sagemaker/latest/dg/smcluster-getting-started-slurm-console.md +++ b//sagemaker/latest/dg/smcluster-getting-started-slurm-console.md @@ -5 +5 @@ -Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and clean resources +Create clusterQuick setupCustom setupDeploy resourcesDelete the cluster and clean resources @@ -7 +7 @@ Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and cl -# Getting started with SageMaker HyperPod using the SageMaker console +# Getting started with SageMaker HyperPod using the SageMaker AI console @@ -9 +9,13 @@ Create your first SageMaker HyperPod cluster with SlurmDelete the cluster and cl -Create your first SageMaker HyperPod cluster using the SageMaker HyperPod console UI. +The following tutorial demonstrates how to create a new SageMaker HyperPod cluster and set it up with Slurm through the SageMaker AI console UI. Following the tutorial, you'll create a HyperPod cluster with three Slurm nodes, `my-controller-group`, `my-login-group`, and `worker-group-1`. + +###### Topics + + * Create cluster + + * Quick setup + + * Custom setup + + * Deploy resources + + * Delete the cluster and clean resources @@ -11 +22,0 @@ Create your first SageMaker HyperPod cluster using the SageMaker HyperPod consol -## Create your first SageMaker HyperPod cluster with Slurm @@ -13 +24,5 @@ Create your first SageMaker HyperPod cluster using the SageMaker HyperPod consol -The following tutorial demonstrates how to create a new SageMaker HyperPod cluster and set it up with Slurm through the SageMaker AI console UI. Following the tutorial, you'll create a HyperPod cluster with three Slurm nodes, `my-controller-group`, `my-login-group`, and `worker-group-1`. + + +## Create cluster + +To navigate to the **SageMaker HyperPod Clusters** page and choose **Slurm** orchestration, follow these steps. @@ -19 +34,38 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust - 3. On the **SageMaker HyperPod Clusters** page, choose **Create cluster**. + 3. On the **SageMaker HyperPod Clusters** page, choose **Create HyperPod cluster**. + + 4. On the **Create HyperPod cluster** drop-down, choose **Orchestrated by Slurm**. + + 5. On the Slurm cluster creation page, you will see two options. Choose the option that best fits your needs. + + 1. **Quick setup** -To get started immediately with default settings, choose **Quick setup**. With this option, SageMaker AI will create new resources such as VPC, subnets, security groups, Amazon S3 bucket, IAM role, and FSx for Lustre in the process of creating your cluster. + + 2. **Custom setup** -To integrate with existing AWS resources or have specific networking, security, or storage requirements, choose **Custom setup**. With this option, you can choose to use the existing resources or create new ones, and you can customize the configuration that best fits your needs. + + + + +## Quick setup + +On the **Quick setup** section, follow these steps to create your HyperPod cluster with Slurm orchestration. + +Specify a name for the new cluster. You can’t change the name after the cluster is created. + +To add an instance group, choose **Add group**. Each instance group can be configured differently, and you can create a heterogeneous cluster that consists of multiple instance groups with various instance types. To deploy a cluster, you must add at least one instance group. + +###### Important + +You can add one instance group at a time. To create multiple instance groups, repeat the process for each instance group. + +Follow these steps to add an instance group. + + 1. For **Instance group type** , choose a type for your instance group. For this tutorial, choose **Controller (head)** for `my-controller-group`, **Login** for `my-login-group`, and **Compute (worker)** for `worker-group-1`. + + 2. For **Name** , specify a name for the instance group. For this tutorial, create three instance groups named `my-controller-group`, `my-login-group`, and `worker-group-1`. + + 3. For **Instance capacity** , choose either on-demand capacity or a training plan to reserve your compute resources. + + 4. For **Instance type** , choose the instance for the instance group. For this tutorial, select `ml.c5.xlarge` for `my-controller-group`, `ml.m5.4xlarge` for `my-login-group`, and `ml.trn1.32xlarge` for `worker-group-1`. + +Ensure that you choose the instance type with sufficient quotas in your account, or request additional quotas by following at [SageMaker HyperPod quotas](./sagemaker-hyperpod-prerequisites.html#sagemaker-hyperpod-prerequisites-quotas). + + 5. For **Instance quantity** , specify an integer not exceeding the instance quota for cluster usage. For this tutorial, enter **1** for all three groups. @@ -21 +73 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust - 4. In **Step 1: Cluster settings** , specify a name for the new cluster. Skip the **Tags** section. + 6. For **Target Availability Zone** , choose the Availability Zone where your instances will be provisioned. The Availability Zone should correspond to the location of your accelerated compute capacity. @@ -23 +75 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust - 5. In **Step 2: Instance groups** , add instance groups. Each instance group can be configured differently, and you can create a heterogeneous cluster that consists of multiple instance groups with various instance types. For lifecycle configuration scripts to run on the instance group during cluster creation, you can start with using the sample lifecycle scripts provided in the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/). + 7. For **Additional storage volume per instance (GB) - optional** , specify an integer between 1 and 16384 to set the size of an additional Elastic Block Store (EBS) volume in gigabytes (GB). The EBS volume is attached to each instance of the instance group. The default mount path for the additional EBS volume is `/opt/sagemaker`. After the cluster is successfully created, you can SSH into the cluster instances (nodes) and verify if the EBS volume is mounted correctly by running the `df -h` command. Attaching an additional EBS volume provides stable, off-instance, and independently persisting storage, as described in the [Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes.html) section in the _Amazon Elastic Block Store User Guide_. @@ -25 +77 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust - 1. For **Instance group name** , specify a name for the instance group. For this tutorial, create three instance groups named `my-controller-group`, `my-login-group`, and `worker-group-1`. + 8. Choose **Add instance group**. @@ -27 +79,41 @@ The following tutorial demonstrates how to create a new SageMaker HyperPod clust - 2. For **Select instance type** , choose the instance for the instance group. For this tutorial, select `ml.c5.xlarge` for `my-controller-group`, `ml.m5.4xlarge` for `my-login-group`, and `ml.trn1.32xlarge` for `worker-group-1`. + + + +This section lists all the default settings for your cluster creation, including all the new AWS resources that will be created during the cluster creation process. Review the default settings. + +## Custom setup + +On the **Custom setup** section, follow these steps to create your HyperPod cluster with Slurm orchestration. + +Specify a name for the new cluster. You can’t change the name after the cluster is created. + +For **Instance recovery** , choose **Automatic -_recommended_** or **None**. + +Configure your network settings for the cluster creation. These settings can't be changed after the cluster is created. + + 1. For **VPC** , choose your own VPC if you already have one that gives SageMaker AI access to your VPC. To create a new VPC, follow the instructions at [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) in the _Amazon Virtual Private Cloud User Guide_. You can leave it as **None** to use the default SageMaker AI VPC. + + 2. For **VPC IPv4 CIDR block** , enter the starting IP of your VPC. + + 3. For **Availability Zones** , choose the Availability Zones (AZ) where HyperPod will create subnets for your cluster. Choose AZs that match the location of your accelerated compute capacity. + + 4. For **Security groups** , create a security group or choose up to five security groups configured with rules to allow inter-resource communication within the VPC. + + + + +To add an instance group, choose **Add group**. Each instance group can be configured differently, and you can create a heterogeneous cluster that consists of multiple instance groups with various instance types. To deploy a cluster, you must add at least one instance group. + +###### Important + +You can add one instance group at a time. To create multiple instance groups, repeat the process for each instance group. + +Follow these steps to add an instance group. + + 1. For **Instance group type** , choose a type for your instance group. For this tutorial, choose **Controller (head)** for `my-controller-group`, **Login** for `my-login-group`, and **Compute (worker)** for `worker-group-1`. + + 2. For **Name** , specify a name for the instance group. For this tutorial, create three instance groups named `my-controller-group`, `my-login-group`, and `worker-group-1`. + + 3. For **Instance capacity** , choose either on-demand capacity or a training plan to reserve your compute resources. + + 4. For **Instance type** , choose the instance for the instance group. For this tutorial, select `ml.c5.xlarge` for `my-controller-group`, `ml.m5.4xlarge` for `my-login-group`, and `ml.trn1.32xlarge` for `worker-group-1`. @@ -31 +123,16 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 3. For **Quantity** , specify an integer not exceeding the instance quota for cluster usage. For this tutorial, enter **1** for all three groups. + 5. For **Instance quantity** , specify an integer not exceeding the instance quota for cluster usage. For this tutorial, enter **1** for all three groups. + + 6. For **Target Availability Zone** , choose the Availability Zone where your instances will be provisioned. The Availability Zone should correspond to the location of your accelerated compute capacity. + + 7. For **Additional storage volume per instance (GB) - optional** , specify an integer between 1 and 16384 to set the size of an additional Elastic Block Store (EBS) volume in gigabytes (GB). The EBS volume is attached to each instance of the instance group. The default mount path for the additional EBS volume is `/opt/sagemaker`. After the cluster is successfully created, you can SSH into the cluster instances (nodes) and verify if the EBS volume is mounted correctly by running the `df -h` command. Attaching an additional EBS volume provides stable, off-instance, and independently persisting storage, as described in the [Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes.html) section in the _Amazon Elastic Block Store User Guide_. + + 8. Choose **Add instance group**. + + + + +You can choose to use the default lifecycle scripts or the custom lifecycle scripts, which will be stored in your Amazon S3 bucket. You can view the default lifecycle scripts in the [Awesome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/tree/main/1.architectures/7.sagemaker-hyperpod-eks/LifecycleScripts). To learn more about the lifecycle scripts, see [Customizing SageMaker HyperPod clusters using lifecycle scripts](./sagemaker-hyperpod-lifecycle-best-practices-slurm.html). + + 1. For **Lifecycle scripts** , choose to use default or custom lifecycle scripts. + + 2. For **S3 bucket for lifecycle scripts** , choose to create a new bucket or use an existing bucket to store the lifecycle scripts. @@ -33 +139,0 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 4. For **S3 path to lifecycle script files** , enter the Amazon S3 path in which your lifecycle scripts are stored. If you don't have lifecycle scripts, go through the following substeps to use the base lifecycle scripts provided by the SageMaker HyperPod service team. @@ -35 +140,0 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 1. Clone the [Awsome Distributed Training GitHub repository](https://github.com/aws-samples/awsome-distributed-training/). @@ -37 +141,0 @@ Ensure that you choose the instance type with sufficient quotas in your account, - git clone https://github.com/aws-samples/awsome-distributed-training/ @@ -39 +143 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 2. Under [`1.architectures/5.sagemaker_hyperpods/LifecycleScripts/base-config`](https://github.com/aws-samples/awsome-distributed-training/tree/main/1.architectures/5.sagemaker-hyperpod/LifecycleScripts/base-config), you can find a set of base lifecycle scripts. To learn more about the lifecycle scripts, see also [Customizing SageMaker HyperPod clusters using lifecycle scripts](./sagemaker-hyperpod-lifecycle-best-practices-slurm.html). +Choose or create an IAM role that allows HyperPod to run and access necessary AWS resources on your behalf. @@ -41 +145 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 3. Write a Slurm configuration file and save it as `provisioning_parameters.json`. In the file, specify basic Slurm configuration parameters to properly assign Slurm nodes to the SageMaker HyperPod cluster instance groups. For example, the `provisioning_parameters.json` should be similar to the following based on the HyperPod cluster instance group configured through the previous steps 5a, 5b, and 5c. +Configure the FSx for Lustre file system to be provisioned on the HyperPod cluster. @@ -43,12 +147 @@ Ensure that you choose the instance type with sufficient quotas in your account, - { - "version": "1.0.0", - "workload_manager": "slurm", - "controller_group": "my-controller-group", - "login_group": "my-login-group", - "worker_groups": [ - { - "instance_group_name": "worker-group-1", - "partition_name": "partition-1" - } - ] - } + 1. For **File system** , choose an existing FSx for Lustre file system, to create a new FSx for Lustre file system, or don't provision an FSx for Lustre file system. @@ -56 +149 @@ Ensure that you choose the instance type with sufficient quotas in your account, - 4. Upload the scripts to your Amazon S3 bucket. Create an S3 bucket with a path in the following format: `s3://sagemaker-`<unique-s3-bucket-name>`/`<lifecycle-script-directory>`/src`. You can create this bucket using the Amazon S3 console. + 2. For **Throughput per unit of storage** , choose the throughput that will be available per TiB of provisioned storage. @@ -58 +151 @@ Ensure that you choose the instance type with sufficient quotas in your account, -###### Note + 3. For **Storage capacity** , enter a capacity value in TB. @@ -60 +153 @@ Ensure that you choose the instance type with sufficient quotas in your account, -You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for SageMaker HyperPod](./sagemaker-hyperpod-prerequisites-iam.html#sagemaker-hyperpod-prerequisites-iam-role-for-hyperpod) with `AmazonSageMakerClusterInstanceRolePolicy` only allows principals to access S3 buckets with this specific prefix. + 4. For **Data compression type** , choose **LZ4** to enable data compression. @@ -62 +155 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 5. For **Directory path to your on-create lifecycle script** , enter the file name of the lifecycle script under **S3 path to lifecycle script files**. + 5. For **Lustre version** , view the value that's recommended for the new file systems. @@ -64 +156,0 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 6. For **IAM role** , choose the IAM role you created using the `AmazonSageMakerClusterInstanceRolePolicy` from the section [IAM role for SageMaker HyperPod](./sagemaker-hyperpod-prerequisites-iam.html#sagemaker-hyperpod-prerequisites-iam-role-for-hyperpod). @@ -66 +157,0 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 7. Under **Advanced configuration** , you can set up the following optional configurations. @@ -68 +158,0 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 1. (Optional) For **Threads per core** , specify `1` for disabling multi-threading and `2` for enabling multi-threading. To find which instance type supports multi-threading, see the reference table of [CPU cores and threads per CPU core per instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/cpu-options-supported-instances-values.html) in the _Amazon Elastic Compute Cloud User Guide_. @@ -70 +160 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 2. (Optional) For **Additional instance storage configs** , specify an integer between 1 and 16384 to set the size of an additional Elastic Block Store (EBS) volume in gigabytes (GB). The EBS volume is attached to each instance of the instance group. The default mount path for the additional EBS volume is `/opt/sagemaker`. After the cluster is succefully created, you can SSH into the cluster instances (nodes) and verify if the EBS volume is mounted correctly by running the `df -h` command. Attaching an additional EBS volume provides stable, off-instance, and independently persisting storage, as described in the [Amazon EBS volumes](https://docs.aws.amazon.com/ebs/latest/userguide/ebs-volumes.html) section in the _Amazon Elastic Block Store User Guide_. +For **Tags -_optional_** , add key and value pairs to the new cluster and manage the cluster as an AWS resource. To learn more, see [Tagging your AWS resources](https://docs.aws.amazon.com/tag-editor/latest/userguide/tagging.html). @@ -72 +162 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 6. In **Step 3: Advanced configuration** , set up network settings within, and in and out of, the cluster. Select your own VPC if you already have one that gives SageMaker AI access to your VPC. If you don't have one but want to create a new VPC, follow the instructions at [Create a VPC](https://docs.aws.amazon.com/vpc/latest/userguide/create-vpc.html) in the _Amazon Virtual Private Cloud User Guide_. You can leave it as **no VPC** to use the default SageMaker AI VPC. +## Deploy resources @@ -74 +164 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 7. In **Step 4: Review and create** , review the configuration you've set from step 1 to 3 and finish submitting the cluster creation request. +After you complete the cluster configurations using either **Quick setup** or **Custom setup** , choose the following option to start resource provisioning and cluster creation. @@ -76 +166 @@ You must prefix `sagemaker-` to the S3 bucket path, because the [IAM role for Sa - 8. The new cluster should appear under **Clusters** in the main pane of the SageMaker HyperPod console. You can check the status of it displayed under the **Status** column. + * **Submit** -SageMaker AI will start provisioning the default configuration resources and creating the cluster.