AWS eks documentation change
Summary
Updated IP address management section from prefix delegation to focus on IP consumption planning for large GPU instances, with recommendations for tuning VPC CNI settings to avoid IP exhaustion.
Security assessment
The changes focus on optimizing IP allocation efficiency and preventing subnet exhaustion during scaling events, which is a resource planning issue rather than a security vulnerability. No evidence of security exploits or vulnerabilities being addressed.
Diff
diff --git a/eks/latest/best-practices/aiml-networking.md b/eks/latest/best-practices/aiml-networking.md index dd55c5062..e3ad71a31 100644 --- a//eks/latest/best-practices/aiml-networking.md +++ b//eks/latest/best-practices/aiml-networking.md @@ -7 +7 @@ -Consider Higher Network Bandwidth or Elastic Fabric Adapter For Applications with High Inter-Node CommunicationIncrease the number of IP addresses available to enable faster pod launch times +Consider Higher Network Bandwidth or Elastic Fabric Adapter For Applications with High Inter-Node CommunicationPlanning for IP Address Consumption on Large GPU Instances @@ -41 +41 @@ Ensure tools like MPI and NCCL are installed in your container image to leverage -## Increase the number of IP addresses available to enable faster pod launch times +## Planning for IP Address Consumption on Large GPU Instances @@ -43 +43 @@ Ensure tools like MPI and NCCL are installed in your container image to leverage -In EKS, each pod needs an IP address from the VPC CIDR block. As your cluster scales with more nodes and pods, you risk IP address exhaustion or slower performance, but enabling prefix delegation can mitigate these issues by pre-allocating IP ranges and reducing EC2 API calls, resulting in faster pod launch times and improved scalability. +By default, the Amazon VPC CNI plugin pre-allocates IP addresses to ensure pods can be scheduled quickly, keeping one full spare ENI attached and populated with IPs. On large instances, this can result in dozens of IPs being reserved per node even when only a few pods are running. @@ -45 +45 @@ In EKS, each pod needs an IP address from the VPC CIDR block. As your cluster sc -Enabling prefix delegation after creating your cluster allows the VPC Container Network Interface (CNI) to assign IP prefixes (/28, each giving 16 IP addresses) to network interfaces on EC2 instances. This means each node can support more pods, reducing the risk of IP shortages. For example, on a `c5.4xlarge` instance, you can support up to 110 pods with prefix delegation. +This mismatch is common in training and inference workloads where pod density per node is low. At cluster scale, especially during autoscaling events that spin up many GPU nodes with few pods each, this can lead to subnet IP exhaustion even though actual IP utilization is low. @@ -47 +47 @@ Enabling prefix delegation after creating your cluster allows the VPC Container -While prefix delegation is crucial for optimizing IP usage in environments with many small pods, AI/ML workloads often use fewer, larger pods (e.g., one pod per GPU). Enabling prefix delegation allows the VPC CNI to pre-allocate a prefix for faster pod startup by maintaining a warm pool. This means IP addresses are readily available, reducing the time needed for pod initialization compared to on-demand allocation in non-prefix mode. In such cases, the IP savings from enabling prefix delegation offers performance benefits for AI/ML workloads. By reducing the number of EC2 API calls required for IP address configuration and pre-allocating IP ranges, using prefix delegation enables faster pod launch times, which is particularly beneficial for quickly scaling AI/ML workloads. +To mitigate this, tune the `WARM_IP_TARGET`, `MINIMUM_IP_TARGET`, and `WARM_ENI_TARGET` variables to match your actual pod density. More info at [VPC CNI’s ENI and IP target settings](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md). @@ -49,6 +49 @@ While prefix delegation is crucial for optimizing IP usage in environments with -To enable prefix delegation: - - - kubectl set env daemonset/aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true - -Ensure proper planning for VPC subnets to avoid IP address exhaustion, especially in large deployments, and manage CIDR blocks to avoid overlaps across VPCs. To learn more, see [Optimizing IP Address Utilization](./ip-opt.html) and [Assign more IP addresses to Amazon EKS nodes with prefixes](https://docs.aws.amazon.com/eks/latest/best-practices/ip-opt.html#_plan_for_growth). +For a full guide on optimizing IP consumption, see [Optimizing IP Address Utilization](https://docs.aws.amazon.com/eks/latest/best-practices/ip-opt.html). @@ -62 +57 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Compute +CPU Inference