AWS eks documentation change
Summary
Updated documentation to add EFA DRA driver (DRANET) as an alternative to EFA device plugin for cross-UltraServer communication, including installation procedures and warnings about coexistence. Added Kubernetes version recommendations and corrected NVIDIA driver repository links.
Security assessment
Changes focus on functionality improvements (topology-aware allocation, device sharing) and installation options without addressing vulnerabilities. The GitHub issue reference (#353) describes an allocation/scheduling problem, not a security flaw. No CVEs, vulnerabilities, or security mitigations are mentioned.
Diff
diff --git a/eks/latest/userguide/ml-eks-nvidia-ultraserver.md b/eks/latest/userguide/ml-eks-nvidia-ultraserver.md index fa2f15154..44a6b6f07 100644 --- a//eks/latest/userguide/ml-eks-nvidia-ultraserver.md +++ b//eks/latest/userguide/ml-eks-nvidia-ultraserver.md @@ -7 +7 @@ -ConsiderationsComponentsProcedureInstall NVIDIA GPU operatorInstall NVIDIA DRA driverInstall the EFA device pluginValidate IMEX over Multi-Node NVLink +ConsiderationsComponentsProcedureInstall NVIDIA GPU operatorInstall NVIDIA DRA driverInstall EFA for cross-UltraServer communicationValidate IMEX over Multi-Node NVLink @@ -55 +55 @@ Components running on node | VPC CNI -EFA device plugin +EFA DRA driver or EFA device plugin @@ -65 +65 @@ The node components in the table above perform the following functions: - * **EFA device plugin** : Allocates EFA devices as secondary networks for pods running on EKS. Responsible for network traffic across P6e-GB200 UltraServers. For multi-node workloads, for GPU-to-GPU within an UltraServer can flow over multi-node NVLink. + * **EFA DRA driver or EFA device plugin** : Allocates EFA devices as secondary networks for pods running on EKS. Responsible for network traffic across P6e-GB200 UltraServers. For multi-node workloads, GPU-to-GPU traffic within an UltraServer can flow over multi-node NVLink. The EFA DRA driver is recommended for Kubernetes 1.34 and later and provides topology-aware allocation and device sharing. The EFA device plugin is supported for all Kubernetes versions. For more information, see [Manage EFA devices on Amazon EKS](./device-management-efa.html). @@ -67 +67 @@ The node components in the table above perform the following functions: - * **NVIDIA Kubernetes device plugin** : Allocates GPUs as devices for pods running on EKS. It is recommended to use the NVIDIA Kubernetes device plugin until the NVIDIA DRA driver GPU allocation functionality graduates from experimental. See the [NVIDIA DRA driver releases](https://github.com/NVIDIA/k8s-dra-driver-gpu/releases) for updated information. + * **NVIDIA Kubernetes device plugin** : Allocates GPUs as devices for pods running on EKS. It is recommended to use the NVIDIA Kubernetes device plugin until the NVIDIA DRA driver GPU allocation functionality graduates from experimental. See the [NVIDIA DRA driver releases](https://github.com/kubernetes-sigs/nvidia-dra-driver-gpu/releases) for updated information. @@ -94 +94,2 @@ NVIDIA DRA Drivers | 25.8.0+ | For ComputeDomain CRDs and IMEX domain manageme -EFA Device Plugin | 0.5.14+ | For cross-UltraServer communication. +EFA DRA driver (DRANET) | Latest | For cross-UltraServer communication with topology-aware allocation. Recommended for Kubernetes 1.34+. +EFA Device Plugin | 0.5.14+ | For cross-UltraServer communication. Supported for all Kubernetes versions. @@ -192 +193 @@ Confirm the DRA resources are available with the following commands. -## Install the EFA device plugin +## Install EFA for cross-UltraServer communication @@ -194 +195,7 @@ Confirm the DRA resources are available with the following commands. -To use EFA communication between UltraServers, you must install the Kubernetes device plugin for EFA. P6e-GB200 instances can be configured with up to [17 network cards](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e) and the primary NCI (index 0) must be of type `interface` and supports up to 100 Gbps of ENA bandwidth. Configure your EFA and ENA interfaces as per your requirements during node provisioning. Review the [EFA configuration for a P6e-GB200 instances AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e) for more details on EFA configuration. +To use EFA communication between UltraServers, install the EFA DRA driver (DRANET) or the EFA device plugin. P6e-GB200 instances can be configured with up to [17 network cards](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e) and the primary NCI (index 0) must be of type `interface` and supports up to 100 Gbps of ENA bandwidth. Configure your EFA and ENA interfaces as per your requirements during node provisioning. Review the [EFA configuration for P6e-GB200 instances AWS documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e) for more details on EFA configuration. + +###### Important + +Do not install the EFA DRA driver and the EFA device plugin on the same node. The two mechanisms cannot coexist on the same node. + +### Option 1: Install the EFA DRA driver (DRANET) @@ -203 +210 @@ To use EFA communication between UltraServers, you must install the Kubernetes d - 2. Install the EFA device plugin for your cluster using the `efa-values.yaml` file you created in the previous step. + 2. Add the EKS Helm chart repository and install the EFA DRA driver. @@ -208,3 +215 @@ To use EFA communication between UltraServers, you must install the Kubernetes d - helm install efa eks/aws-efa-k8s-device-plugin -n kube-system \ - --version="0.5.14" \ - -f efa-values.yaml + helm install aws-dranet eks/aws-dranet --namespace kube-system -f efa-values.yaml @@ -212 +217 @@ To use EFA communication between UltraServers, you must install the Kubernetes d -As an example, if you configured your instances with 1 efa-only interface in each [NCI group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e), when describing a node, it is expected to see 4 allocatable EFA devices per node. + 3. Verify that the DRANET DaemonSet is running. @@ -214 +219 @@ As an example, if you configured your instances with 1 efa-only interface in eac - kubectl describe node/<gb200-node-name> + kubectl get daemonset -n kube-system aws-dranet @@ -216,6 +221,47 @@ As an example, if you configured your instances with 1 efa-only interface in eac - Capacity: - ... - vpc.amazonaws.com/efa: 4 - Allocatable: - ... - vpc.amazonaws.com/efa: 4 + NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE + aws-dranet 2 2 2 2 2 <none> 60s + + 4. Verify that the `DeviceClass` and `ResourceSlice` objects are available. + + kubectl get deviceclass efa.networking.k8s.aws + + NAME AGE + efa.networking.k8s.aws 60s + + kubectl get resourceslices -l resource.k8s.io/driver=dra.net + + + + +For more information on using the EFA DRA driver, including topology-aware allocation and device sharing, see [Manage EFA devices on Amazon EKS](./device-management-efa.html). + +### Option 2: Install the EFA device plugin + + 1. Create a Helm values file named `efa-values.yaml` with the following configuration. + + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + + 2. Add the EKS Helm chart repository and install the EFA device plugin. + + helm repo add eks https://aws.github.io/eks-charts + helm repo update + + helm install efa eks/aws-efa-k8s-device-plugin -n kube-system -f efa-values.yaml + + 3. Verify the EFA device plugin DaemonSet is running. + + kubectl get daemonset -n kube-system aws-efa-k8s-device-plugin-daemonset + + NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE + aws-efa-k8s-device-plugin-daemonset 2 2 2 2 2 <none> 60s + + 4. Verify that your nodes have allocatable EFA devices. As an example, if you configured your instances with 1 efa-only interface in each [NCI group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html#efa-for-p6e), it is expected to see 4 allocatable EFA devices per node. + + kubectl get nodes "-o=custom-columns=NAME:.metadata.name,EFA:.status.allocatable.vpc\.amazonaws\.com/efa" + + NAME EFA + ip-192-168-11-225.us-west-2.compute.internal 4 + ip-192-168-24-96.us-west-2.compute.internal 4 @@ -234 +280,8 @@ For a multi-node NVLINK NCCL test and other micro-benchmarks review the [awesome - 2. Create a Helm values file named `nvbandwidth-test-job.yaml` that defines the test manifest. Note the `nvidia.com/gpu.clique` pod affinity to schedule the workers in the same NVLink domain which has Multi-Node NVLink reachability. The sample below runs a multi-node device-to-device CE Read memcpy test using cuMemcpyAsync and prints the results in the logs. + 2. Create a file named `nvbandwidth-test-job.yaml` that defines the test manifest. Note the `nvidia.com/gpu.clique` pod affinity to schedule the workers in the same NVLink domain which has Multi-Node NVLink reachability. The sample below runs a multi-node device-to-device CE Read memcpy test using cuMemcpyAsync and prints the results in the logs. + +As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec.numNodes` can be set to `0` in the ComputeDomain definition. Review the latest [NVIDIA DRA Driver release notes](https://github.com/kubernetes-sigs/nvidia-dra-driver-gpu/releases) for updates. + +There can be only one ComputeDomain (IMEX channel) per node. Do not change the `allocationMode` to `All` for the ComputeDomain resource, as it can prevent the ComputeDomain and Pods accessing that ComputeDomain from being allocated and scheduled correctly. For more information, see [NVIDIA DRA driver issue #353](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu/issues/353). + + + @@ -236 +288,0 @@ For a multi-node NVLINK NCCL test and other micro-benchmarks review the [awesome -As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec.numNodes` can be set to `0` in the ComputeDomain definition. Review the latest [NVIDIA DRA Driver release notes](https://github.com/NVIDIA/k8s-dra-driver-gpu) for updates. @@ -346 +398,4 @@ As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec. - 3. Create the ComputeDomain and start the job with the following command. +\+ . Create the ComputeDomain and start the job with the following command. + ++ + @@ -350 +405,4 @@ As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec. - 4. ComputeDomain creation, you can see the workload’s ComputeDomain has two nodes: +\+ . ComputeDomain creation, you can see the workload’s ComputeDomain has two nodes: + ++ + @@ -353,0 +412,3 @@ As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec. ++ + + @@ -364 +425,4 @@ As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec. - 5. Review the results of the job with the following command. +\+ . Review the results of the job with the following command. + ++ + @@ -368 +432,4 @@ As of NVIDIA DRA Driver version `v25.8.0` ComputeDomains are elastic and `.spec. -A successful test shows bandwidth statistics in GB/s for the multi-node memcpy test. An example of a successful test output is shown below. +\+ A successful test shows bandwidth statistics in GB/s for the multi-node memcpy test. An example of a successful test output is shown below. + ++ + @@ -405,3 +472 @@ A successful test shows bandwidth statistics in GB/s for the multi-node memcpy t - 6. When the test is complete, delete it with the following command. - - kubectl delete -f nvbandwidth-test-job.yaml +\+ . When the test is complete, delete it with the following command. @@ -408,0 +474 @@ A successful test shows bandwidth statistics in GB/s for the multi-node memcpy t ++ @@ -410,0 +477 @@ A successful test shows bandwidth statistics in GB/s for the multi-node memcpy t + kubectl delete -f nvbandwidth-test-job.yaml