AWS sagemaker documentation change
Summary
Added warning about NVMe CSI driver conflicts and clarified NVMe storage usage guidance
Security assessment
The change addresses potential I/O failures from CSI driver conflicts but doesn't reference any security vulnerability, exploit, or security incident. It focuses on operational reliability of storage systems without security implications.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-eks-prerequisites.md b/sagemaker/latest/dg/sagemaker-hyperpod-eks-prerequisites.md index 2b01cb1b2..97c06b1f0 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-eks-prerequisites.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-eks-prerequisites.md @@ -55,0 +56,4 @@ When provisioning your Amazon EKS cluster, consider the following: +###### Important + +Do not install a CSI driver that manages NVMe instance storage (sometimes called a Local Instance Storage CSI driver) on HyperPod nodes that use the SageMaker HyperPod AMI. The AMI configures an LVM volume group (`vg.01`) on the NVMe instance store at boot, before the CSI driver starts, and a CSI driver that targets the same NVMe devices will conflict with this configuration and cause I/O failures. The Amazon EBS CSI driver, the Amazon FSx CSI driver, the Amazon EFS CSI driver, and the Mountpoint for Amazon S3 CSI driver are not affected because they do not manage local NVMe instance storage. For details and remediation steps, see the Considerations section in [Using the Amazon EBS CSI driver on SageMaker HyperPod EKS clusters](./sagemaker-hyperpod-eks-ebs.html). + @@ -63 +67 @@ When provisioning your Amazon EKS cluster, consider the following: - * You can provision and mount additional Amazon EBS volumes on SageMaker HyperPod nodes using two approaches: use [_InstanceStorageConfigs_](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ClusterInstanceGroupSpecification.html#sagemaker-Type-ClusterInstanceGroupSpecification-InstanceStorageConfigs) for cluster-level volume provisioning (available when creating or updating instance groups), or use the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver for dynamic pod-level volume management. With [_InstanceStorageConfigs_](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ClusterInstanceGroupSpecification.html#sagemaker-Type-ClusterInstanceGroupSpecification-InstanceStorageConfigs), set the [local path](https://kubernetes.io/docs/concepts/storage/volumes/#local) to `/opt/sagemaker` to properly mount the volumes to your Amazon EKS pods. For information about how to deploy the [Amazon EBS CSI](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) controller on HyperPod nodes, see [Using the Amazon EBS CSI driver on SageMaker HyperPod EKS clusters](./sagemaker-hyperpod-eks-ebs.html). + * You can provision and mount additional Amazon EBS volumes on SageMaker HyperPod nodes using two approaches: use [_InstanceStorageConfigs_](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ClusterInstanceGroupSpecification.html#sagemaker-Type-ClusterInstanceGroupSpecification-InstanceStorageConfigs) for cluster-level volume provisioning (available when creating or updating instance groups), or use the Amazon Elastic Block Store (Amazon EBS) Container Storage Interface (CSI) driver for dynamic pod-level volume management. With [_InstanceStorageConfigs_](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ClusterInstanceGroupSpecification.html#sagemaker-Type-ClusterInstanceGroupSpecification-InstanceStorageConfigs), set the [local path](https://kubernetes.io/docs/concepts/storage/volumes/#local) to `/opt/sagemaker` to properly mount the volumes to your Amazon EKS pods. For information about how to deploy the [Amazon EBS CSI](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) controller on HyperPod nodes, see [Using the Amazon EBS CSI driver on SageMaker HyperPod EKS clusters](./sagemaker-hyperpod-eks-ebs.html). To use NVMe instance storage on HyperPod, rely on the AMI's built-in LVM configuration through `/opt/sagemaker` rather than installing a separate NVMe-targeting CSI driver. @@ -65 +69 @@ When provisioning your Amazon EKS cluster, consider the following: - * If you use instance-type labels for defining scheduling constraints, ensure that you use the SageMaker AI ML instance types prefixed with `ml.`. For example, for P5 instances, use `ml.p5.48xlarge` instead of `p5.48xlarge`. + * If you use instance-type labels to define scheduling constraints, ensure that you use the SageMaker AI ML instance types prefixed with `ml.`. For example, for P5 instances, use `ml.p5.48xlarge` instead of `p5.48xlarge`.