AWS eks documentation change
Summary
Comprehensive update to EFA documentation introducing EFA DRA driver (DRANET), comparing it with existing EFA device plugin, adding detailed configuration examples for Karpenter and managed node groups, explaining topology-aware allocation with GPUs/Neuron devices, and expanding installation procedures.
Security assessment
The changes focus on performance optimization, new resource allocation mechanisms, and configuration best practices for EFA devices. There is no evidence of security vulnerability fixes or security incident response. The security group configuration mentioned is standard networking practice for EFA and not addressing a specific security flaw.
Diff
diff --git a/eks/latest/userguide/device-management-efa.md b/eks/latest/userguide/device-management-efa.md index 228c7d3c6..4a8ac503e 100644 --- a//eks/latest/userguide/device-management-efa.md +++ b//eks/latest/userguide/device-management-efa.md @@ -7 +7 @@ -Creating EKS nodes with EFA interfacesUsing EKS-optimized AMIs with EFAInstall the EFA Kubernetes device plugin +EFA DRA driver vs. EFA device pluginCreating EKS nodes with EFA interfacesKarpenterEKS managed node groups and self-managed nodesUsing EKS-optimized AMIs with EFAConserving IP address allocationInstall the EFA DRA driver (DRANET)Topology-aware EFA and GPU/Neuron device allocationShare EFA devices between multiple PodsInstall the EFA Kubernetes device plugin @@ -15 +15 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -[Elastic Fabric Adapter](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) (EFA) is a network device for Amazon EC2 instances that enables high-performance inter-node communication for machine learning training and High Performance Computing (HPC) workloads. Amazon EKS supports the _EFA device plugin_ for managing EFA devices in EKS clusters. +[Elastic Fabric Adapter](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) (EFA) is a network device for Amazon EC2 instances that enables high-performance inter-node communication and RDMA (Remote Direct Memory Access) for artificial intelligence, machine learning, and High Performance Computing (HPC) workloads. Amazon EKS supports two mechanisms for managing EFA devices in EKS clusters: the _EFA Dynamic Resource Allocation (DRA) driver (DRANET)_ and the _EFA device plugin_. @@ -17 +17,5 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -## Creating EKS nodes with EFA interfaces +It’s recommended to use the EFA DRA driver (DRANET) for new deployments on EKS clusters running Kubernetes version 1.34 or later with EKS managed node groups or self-managed node groups. The EFA DRA driver makes it possible for you to configure topology-aware allocation that pairs EFA interfaces with their topologically-local GPUs or Neuron devices, and supports device sharing between Pods. + +The EFA DRA driver is not supported with Karpenter or EKS Auto Mode. Use the EFA device plugin with Karpenter and EKS Auto Mode. The EFA device plugin also remains supported for EKS managed node groups and self-managed nodes. + +## EFA DRA driver vs. EFA device plugin @@ -19 +23,11 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -When you create EKS nodes with EFA interfaces, the EFA interfaces are attached during instance bootstrap. If you need to customize the per-device EFA configuration or use [placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) for the EFA-enabled EC2 instances, it’s recommended to use EKS managed node groups or EKS self-managed node groups. You can pass configuration for each network interface with [launch templates](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html). +Feature | EFA DRA driver | EFA device plugin +---|---|--- +Minimum Kubernetes version | 1.34 | All EKS-supported Kubernetes versions +EKS Compute | Managed node groups, self-managed nodes | EKS Auto Mode, Karpenter, managed node groups, self-managed nodes +EKS-optimized AMIs | AL2023 (NVIDIA, Neuron), Bottlerocket | AL2023 (NVIDIA, Neuron), Bottlerocket +Device advertisement | Rich attributes via `ResourceSlice` objects including device type, topology, and PCIe locality | Integer count of `vpc.amazonaws.com/efa` extended resources +GPU-EFA affinity | DRA-native topology-awareness | Automatic topology-awareness (EKS-optimized AL2023 AMIs only) +Neuron-EFA affinity | DRA-native topology-awareness | Automatic topology-awareness (EKS-optimized AL2023 AMIs only) +Device sharing | Multiple Pods can share the same EFA device through shared `ResourceClaim` references | Not supported. Each EFA device is exclusively allocated to one Pod. + +## Creating EKS nodes with EFA interfaces @@ -21 +35 @@ When you create EKS nodes with EFA interfaces, the EFA interfaces are attached d -When using EKS Auto Mode or Karpenter with dynamic provisioning, instances created for Pods requesting `vpc.amazonaws.com/efa` have all interfaces configured with interface type `EFA`. Static capacity provisioning in EKS Auto Mode and Karpenter does not currently support per-device EFA configuration. EKS Auto Mode and Karpenter do not currently support placement groups. +When you create EKS nodes with EFA interfaces, the EFA interfaces are attached to the instance during instance provisioning. You can customize the per-device EFA configuration and use [placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) with Karpenter, EKS managed node groups, or EKS self-managed node groups. With Karpenter, you pass configuration for each network interface via the `NodeClass`. With EKS managed node groups or self-managed nodes, you pass configuration for each network interface with [launch templates](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html). EKS Auto Mode support for per-device EFA configuration and placement groups is coming soon. @@ -24,0 +39,121 @@ When using [eksctl](./install-kubectl.html#eksctl-install-update) for provisioni +The following examples show how to configure `NodeClass` and launch templates with EFA interfaces. This is useful to customize the interfaces used for EFA vs standard IP-based traffic. For information on the number of EFA interfaces supported by each instance type and how to configure them for maximum network bandwidth, see [Maximize network bandwidth for EFA-enabled instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html) in the _Amazon EC2 User Guide_. + +## Karpenter + +Each entry in `networkInterfaces` specifies a `networkCardIndex`, `deviceIndex`, and `interfaceType`. The `interfaceType` can be `interface` for standard network interfaces or `efa-only` for EFA interfaces that are dedicated to RDMA traffic and do not have IP addresses assigned. When `networkInterfaces` is configured, instances launched by the `NodePool` referencing the `NodeClass` use this configuration regardless of whether Pods request `vpc.amazonaws.com/efa` resources. + +When using Karpenter without specifying `networkInterfaces` in your `NodeClass`, instances created for Pods requesting `vpc.amazonaws.com/efa` have all interfaces configured with interface type `EFA`. + +The `networkInterfaces` configuration for `EC2NodeClass` was added in Karpenter v1.11. The following example shows an `EC2NodeClass` configured for a P6-B200 instance with 1 ENA interface and 8 EFA-only interfaces. + + + apiVersion: karpenter.k8s.aws/v1 + kind: EC2NodeClass + metadata: + name: efa-node-class + spec: + networkInterfaces: + - networkCardIndex: 0 + deviceIndex: 0 + interfaceType: interface + - networkCardIndex: 0 + deviceIndex: 1 + interfaceType: efa-only + - networkCardIndex: 1 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 2 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 3 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 4 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 5 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 6 + deviceIndex: 0 + interfaceType: efa-only + - networkCardIndex: 7 + deviceIndex: 0 + interfaceType: efa-only + +## EKS managed node groups and self-managed nodes + +With EKS managed node groups or self-managed nodes, you pass configuration for each network interface with [launch templates](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html). + +The following example shows a launch template configured for a P6-B200 instance with 1 ENA interface and 8 EFA-only interfaces. The primary network interface (network card 0, device index 0) uses a standard `interface` type for IP traffic, while additional interfaces use `efa-only` for dedicated RDMA traffic. Adjust the number of `efa-only` interfaces based on your instance type. For the number of EFA interfaces supported by each instance type, see [Maximize network bandwidth for EFA-enabled instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html) in the _Amazon EC2 User Guide_. + +Replace ` _security-group-id_ ` with your values. The security group must allow all inbound and outbound traffic to and from itself to enable EFA OS-bypass functionality. For more information, see [Step 1: Prepare an EFA-enabled security group](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-security) in the _Amazon EC2 User Guide_. + +###### Important + +Do not specify `SubnetId` in the launch template when using EKS managed node groups. EKS requires that all subnets are specified through the `CreateNodegroup` API and rejects launch templates that include subnet configuration. + + + { + "LaunchTemplateName": "efa-launch-template", + "LaunchTemplateData": { + "InstanceType": "p6-b200.48xlarge", + "NetworkInterfaces": [ + { + "NetworkCardIndex": 0, + "DeviceIndex": 0, + "InterfaceType": "interface", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 0, + "DeviceIndex": 1, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 1, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 2, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 3, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 4, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 5, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 6, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + }, + { + "NetworkCardIndex": 7, + "DeviceIndex": 0, + "InterfaceType": "efa-only", + "Groups": ["security-group-id"] + } + ] + } + } + @@ -27 +162,242 @@ When using [eksctl](./install-kubectl.html#eksctl-install-update) for provisioni -The EKS-optimized AL2023 accelerated AMIs (NVIDIA and Neuron) and all Bottlerocket AMIs include the host-level components required to use EFA. The EKS AL2023 and Bottlerocket AMIs do not include the EFA device plugin, and the device plugin must be installed separately on your cluster before deploying workloads that use EFA. +The EKS-optimized AL2023 accelerated AMIs (NVIDIA and Neuron) and all Bottlerocket AMIs include the host-level components required to use EFA, specifically the components installed by the [aws-efa-installer](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable). The EKS AL2023 and Bottlerocket AMIs **do not include** the EFA DRA driver or EFA device plugin, and these must be installed separately on your cluster before deploying workloads. + +## Conserving IP address allocation + +EFA-enabled instances such as `p5.48xlarge` and `p6-b200.48xlarge` support many network interfaces. By default, the Amazon VPC CNI allocates IP addresses across all IP-enabled attached ENIs, which can consume a large number of IP addresses from your subnet even when those addresses are not actively used by Pods. On instances with dozens of network interfaces, this can quickly exhaust your subnet’s available IP space. + +To reduce IP address consumption on EFA-enabled nodes, configure your network interfaces to use `efa-only` for all interfaces except the primary. EFA-only interfaces are dedicated to RDMA traffic and do not have IP addresses assigned, so they do not consume addresses from your subnet. For example configurations, see Karpenter and EKS managed node groups and self-managed nodes. For the recommended interface layout for each instance type, see [Maximize network bandwidth for EFA-enabled instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-acc-inst-types.html) in the _Amazon EC2 User Guide_. + +In addition to using `efa-only` interfaces, you can configure the Amazon VPC CNI to limit the number of warm (pre-allocated) IP addresses and ENIs. By default, the VPC CNI pre-allocates a warm pool of ENIs and IP addresses for faster Pod startup, but on large instances this can reserve hundreds of unused IP addresses. Set the `WARM_IP_TARGET` and `WARM_ENI_TARGET` environment variables on the `aws-node` DaemonSet to control how many spare IP addresses and ENIs the CNI maintains. For more information on these settings, see [Amazon VPC CNI best practices](https://docs.aws.amazon.com/eks/latest/best-practices/vpc-cni.html#_overview). + +###### Note + +The `WARM_ENI_TARGET` and `WARM_IP_TARGET` settings are cluster-wide and apply to all nodes managed by the VPC CNI. There is currently no way to set different values per node group or instance type. If you need more granular control of these settings, provide feedback on [containers-roadmap issue #1834](https://github.com/aws/containers-roadmap/issues/1834). + +## Install the EFA DRA driver (DRANET) + +The EFA DRA driver is built in the upstream [DRANET](https://github.com/kubernetes-sigs/dranet) project, which provides cloud-aware network device management for Kubernetes DRA. _EFA DRA driver_ and _DRANET_ are used interchangeably throughout this documentation and refer to the same tool. + +The EFA DRA driver advertises EFA devices as `ResourceSlice` objects with the driver name `dra.net` and the `DeviceClass` name `efa.networking.k8s.aws`. The EFA DRA driver runs as a DaemonSet on each node and automatically discovers EFA devices. + +### Prerequisites + + * An Amazon EKS cluster running Kubernetes version 1.34 or later with EKS managed node groups or self-managed node groups. + + * Nodes with EFA-enabled Amazon EC2 instance types. For a list of supported instance types, see [Supported instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-instance-types) in the _Amazon EC2 User Guide_. + + * Nodes with host-level components installed for EFA, see [Install the EFA software](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start.html#efa-start-enable) for more information. The EKS-optimized AL2023 NVIDIA and Neuron AMIs, and the Bottlerocket AMIs include the EFA host-level components. + + * Helm installed in your command-line environment, see the [Setup Helm instructions](./helm.html) for more information. + + * `kubectl` configured to communicate with your cluster, see [Install or update kubectl](./install-kubectl.html#kubectl-install-update) for more information. + + + + +### Procedure + +###### Important + +Do not install the EFA DRA driver on nodes where the EFA device plugin is running. The two mechanisms cannot coexist on the same node. See upstream Kubernetes [KEP-5004](https://github.com/kubernetes/enhancements/issues/5004) for updates. + + 1. Add the EKS Helm chart repository. +