AWS eks documentation change
Summary
Updated networking best practices documentation with details about Kubernetes topology-aware routing evolution, added Traffic Distribution feature guidance, and expanded operational considerations for cross-AZ traffic management
Security assessment
The changes focus on cost optimization and network efficiency through topology-aware routing and traffic distribution features. While they discuss workload distribution and AZ awareness, there is no mention of security vulnerabilities, access controls, or data protection mechanisms. The updates primarily address operational efficiency and cost management rather than security concerns.
Diff
diff --git a/eks/latest/best-practices/cost-opt-networking.md b/eks/latest/best-practices/cost-opt-networking.md index ab0b629e8..061af5083 100644 --- a//eks/latest/best-practices/cost-opt-networking.md +++ b//eks/latest/best-practices/cost-opt-networking.md @@ -21 +21 @@ Depending on your setup, network communication and data transfer between Pods ca -Frequent egress cross-zone traffic (traffic distributed between AZs) can have a major impact on your network-related costs. Below are some strategies on how to control the amount of cross-zone traffic between Pods in your EKS cluster. +The Kubernetes project early on began developing topology-aware constructs including labels like kubernetes.io/hostname, topology.kubernetes.io/region, and topology.kubernetes.io/zone assigned to nodes to enable features such as workload distribution across failure domains and topology-aware volume provisioners. Having graduated in Kubernetes 1.17, the labels were also leveraged to enable topology-aware routing capabilities for Pod to Pod communication. @@ -23 +23 @@ Frequent egress cross-zone traffic (traffic distributed between AZs) can have a -_If you want granular visibility into the amount of cross-zone traffic between Pods in your cluster (such as the amount of data transferred in bytes),[ refer to this post](https://aws.amazon.com/blogs/containers/getting-visibility-into-your-amazon-eks-cross-az-pod-to-pod-network-bytes/)._ +Below are some strategies on how to control the amount of cross-AZ traffic between Pods in your EKS cluster to reduce costs and minimize latency. @@ -25 +25 @@ _If you want granular visibility into the amount of cross-zone traffic between P -**Using Topology Aware Routing (formerly known as Topology Aware Hints)** +_If you want granular visibility into the amount of cross-AZ traffic between Pods in your cluster (such as the amount of data transferred in bytes),[ refer to this post](https://aws.amazon.com/blogs/containers/getting-visibility-into-your-amazon-eks-cross-az-pod-to-pod-network-bytes/)._ @@ -29 +29,5 @@ _If you want granular visibility into the amount of cross-zone traffic between P -When using topology aware routing, it’s important to understand how Services, EndpointSlices and the `kube-proxy` work together when routing traffic. As the diagram above depicts, Services are the stable network abstraction layer that receive traffic destined for your Pods. When a Service is created, multiple EndpointSlices are created. Each EndpointSlice has a list of endpoints containing a subset of Pod addresses along with the nodes they’re running on and any additional topology information. `kube-proxy` is a daemonset that runs on every node in your cluster and also fulfills a role of internal routing, but it does so based on what it consumes from the created EndpointSlices. +As the diagram above depicts, Services are the stable network abstraction layer that receive traffic destined for your Pods. When a Service is created, multiple EndpointSlices are created. Each EndpointSlice has a list of endpoints containing a subset of Pod addresses along with the nodes they’re running on and any additional topology information. When using the Amazon VPC CNI, kube-proxy, a daemonset running on every node, maintains network rules to enable Pod communication and Service discovery (alternative eBPF-based CNIs may not use kube-proxy but provide equivalent behavior). It fulfills the role of internal routing, but it does so based on what it consumes from the created EndpointSlices. + +On EKS, kube-proxy primarily uses iptables NAT rules (or [IPVS](https://docs.aws.amazon.com/eks/latest/best-practices/ipvs.html), [NFTables](https://kubernetes.io/blog/2025/02/28/nftables-kube-proxy/) as alternatives) for traffic distribution across all pods in the cluster, regardless of their node or AZ placement. This default distribution can lead to cross-AZ traffic routing, potentially causing increased latency for sensitive applications and inter-AZ data transfer charges in large deployments. + +**Using Topology Aware Routing (formerly known as Topology Aware Hints)** @@ -65 +69,42 @@ The screenshot below shows the result of the EndpointSlice controller having suc -It’s important to note that topology aware routing is still in **beta**. Also, this feature is more predictable when workloads are widely and evenly distributed across the cluster topology. Therefore, it is highly recommended to use it in conjunction with scheduling constraints that increase the availability of an application such as [pod topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/). +It’s important to note that topology aware routing is still in beta. This feature performs more predictably with evenly distributed workloads across the cluster topology, as the controller allocates endpoints proportionally across zones but may skip hint assignments when node resources in a zone are too imbalanced to avoid excessive overload. Therefore, it is highly recommended to use it in conjunction with scheduling constraints that increase the availability of an application such as [pod topology spread constraints](https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/). Note that hints may also not be assigned when capacity fluctuates across zones, such as when using [Amazon EC2 Spot Instances](https://aws.amazon.com/ec2/spot/), as interruptions or replacements are not detected in real-time when calculating the proportional distribution. + +**Using Traffic Distribution** + +Introduced in Kubernetes 1.30 and made generally available in 1.33, [Traffic Distribution](https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-distribution) offers a simpler alternative to Topology Aware Routing for same-zone traffic preference. While Topology Aware Routing attempts to use an intelligent approach to traffic routing to avoid overloading endpoints, it resulted in unpredictable behavior. Traffic Distribution prioritizes predictability instead. The PreferClose option directs kube-proxy to create rules that route traffic to same-zone endpoints first based on the zonal _hint_ set by the EndpointSlice Controller. When no same-zone endpoints are available, it falls back to distributing traffic across any cluster endpoint for the Service. This feature is designed for workloads that accept the tradeoff of optimizing for proximity rather than the attempted even distribution of load that Topology Aware Routing provides. + +Below is a code snippet on how to enable _traffic distribution_ for a Service. + + + apiVersion: v1 + kind: Service + metadata: + name: orders-service + namespace: ecommerce + spec: + trafficDistribution: PreferClose + selector: + app: orders + type: ClusterIP + ports: + + * protocol: TCP + port: 3003 + targetPort: 3003 + +When enabling Traffic Distribution, a common challenge emerges: endpoints within a single AZ can become overloaded if most traffic originates from that same zone. This overloading can create significant issues: + + * A single Horizontal Pod Autoscaler (HPA) managing a multi-AZ deployment may respond by scaling out pods across different AZs. However, this action fails to effectively address the increased load in the affected zone. + + * This situation in turn can lead to resource inefficiency. When cluster autoscalers like Karpenter detect the pod scale-out across different AZs, they may provision additional nodes in the unaffected AZs, resulting in unnecessary resource allocation. + + + + +To overcome this challenge: + + * Create separate deployments per zone which would have their own HPAs to scale independent of one another. + + * Leverage Topology Spread Constraints to ensure workload distribution across the cluster, which helps prevent endpoint overloads in high-traffic zones. + + +