AWS sagemaker documentation change
Summary
Added documentation about restricted instance groups (RIG) configuration, including prerequisites, dependency installation script details, and FSx for Lustre synchronization requirements. Removed CloudWatch logging setup reference and added configuration parameters for RestrictedInstanceGroups.
Security assessment
The changes introduce documentation about security-hardened instance groups (RIG) with specific network configurations (security groups/subnets) and data synchronization requirements. While these are security-related features, there's no evidence of addressing a specific vulnerability. The modifications to CNI/CoreDNS configurations and taint tolerations improve isolation but are presented as new capabilities rather than vulnerability fixes.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-cli-command-create-cluster.md b/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-cli-command-create-cluster.md index 745616491..39209caed 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-cli-command-create-cluster.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-cli-command-create-cluster.md @@ -13 +13 @@ Learn how to create SageMaker HyperPod clusters orchestrated by Amazon EKS using - 2. Install the Helm chart as instructed in [Installing packages on the Amazon EKS cluster using Helm](./sagemaker-hyperpod-eks-install-packages-using-helm-chart.html). + 2. Install the Helm chart as instructed in [Installing packages on the Amazon EKS cluster using Helm](./sagemaker-hyperpod-eks-install-packages-using-helm-chart.html). If you create a [Creating a HyperPod EKS cluster with restricted instance group (RIG)](./nova-hp-cluster.html), you will need a separate Helm chart. @@ -17 +17 @@ Learn how to create SageMaker HyperPod clusters orchestrated by Amazon EKS using -For a quick start, download the sample script [`on_create.sh`](https://github.com/aws-samples/awsome-distributed-training/blob/main/1.architectures/7.sagemaker-hyperpod-eks/LifecycleScripts/base-config/on_create.sh) from the AWSome Distributed Training GitHub repository, and upload it to the S3 bucket. This script sets up the logging file `/var/log/provision/provisioning.log` required for CloudWatch to gather logs from Pod containers. You can also include additional setup instructions, a series of setup scripts, or commands to be executed during the HyperPod cluster provisioning stage. +For a quick start, download the sample script [`on_create.sh`](https://github.com/aws-samples/awsome-distributed-training/blob/main/1.architectures/7.sagemaker-hyperpod-eks/LifecycleScripts/base-config/on_create.sh) from the AWSome Distributed Training GitHub repository, and upload it to the S3 bucket. You can also include additional setup instructions, a series of setup scripts, or commands to be executed during the HyperPod cluster provisioning stage. @@ -22,0 +23,18 @@ If you create an [IAM role for SageMaker HyperPod](./sagemaker-hyperpod-prerequi +If you create a restricted instance group, you don't need to download and run the lifecycle script. Instead, you need to run `install_rig_dependencies.sh`. + +The prerequisites to run the `install_rig_dependencies.sh` script include: + + * AWS Node (CNI) and CoreDNS should both be enabled. These are standard EKS add-ons that are not managed by the standard SageMaker HyperPod Helm, but can be easily enabled in the EKS console under Add-ons. + + * The standard SageMaker HyperPod Helm chart should be installed before running this script. + +The `install_rig_dependencies.sh` script performs the following actions. + + * `aws-node` (CNI): New `rig-aws-node` Daemonset created; existing `aws-node` patched to avoid RIG nodes. + + * `coredns`: Converted to Daemonset for RIGs to support multi-RIG use and prevent overloading. + + * training-operators: Updated with RIG Worker taint tolerations and nodeAffinity favoring non-RIG instances. + + * Elastic Fabric Adapter (EFA): Updated to tolerate RIG worker taint and use correct container images for each Region. + @@ -44,0 +63,45 @@ Ensure that your SageMaker HyperPod cluster is deployed within the same Virtual + "RestrictedInstanceGroups": [ + { + "EnvironmentConfig": { + "FSxLustreConfig": { + "PerUnitStorageThroughput": number, + "SizeInGiB": number + } + }, + "ExecutionRole": "string", + "InstanceCount": number, + "InstanceGroupName": "string", + "InstanceStorageConfigs": [ + { ... } + ], + "InstanceType": "string", + "OnStartDeepHealthChecks": [ "string" ], + "OverrideVpcConfig": { + "SecurityGroupIds": [ "string" ], + "Subnets": [ "string" ] + }, + "ScheduledUpdateConfig": { + "DeploymentConfig": { + "AutoRollbackConfiguration": [ + { + "AlarmName": "string" + } + ], + "RollingUpdatePolicy": { + "MaximumBatchSize": { + "Type": "string", + "Value": number + }, + "RollbackMaximumBatchSize": { + "Type": "string", + "Value": number + } + }, + "WaitIntervalInSeconds": number + }, + "ScheduleExpression": "string" + }, + "ThreadsPerCore": number, + "TrainingPlanArn": "string" + } + ], @@ -85,0 +149,6 @@ This should return the ARN of the new cluster. +###### Important + +You can use the [update-cluster](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-cluster.html) operation to remove a restricted instance group (RIG). When a RIG is scaled down to 0, the FSx for Lustre file system won't be deleted. To completely remove the FSx for Lustre file system, you must remove the RIG entirely. + +Removing a RIG will not delete any artifacts stored in the service-managed Amazon S3 bucket. However, you should ensure all artifacts in the FSx for Lustre file system are fully synchronized to Amazon S3 before removal. We recommend waiting at least 30 minutes after job completion to ensure complete synchronization of all artifacts from the FSx for Lustre file system to the service-managed Amazon S3 bucket. +