AWS eks medium security documentation change
Summary
Expanded networking documentation for EKS hybrid nodes with detailed routable/unroutable pod network guidance, security group configuration requirements, and operational warnings
Security assessment
Added explicit warnings about security group rule limits (60 ingress max) and CIDR cleanup responsibilities. These changes address potential security misconfigurations where exceeding rule limits could prevent critical security rules from being applied, and failure to clean up CIDRs could leave stale network access. The documentation now emphasizes security-conscious network configuration through NAT/egress masquerade requirements for unroutable networks.
Diff
diff --git a/eks/latest/userguide/hybrid-nodes-networking.md b/eks/latest/userguide/hybrid-nodes-networking.md index 3c501e0e5..44dbe22ac 100644 --- a//eks/latest/userguide/hybrid-nodes-networking.md +++ b//eks/latest/userguide/hybrid-nodes-networking.md @@ -19 +19 @@ This topic provides an overview of the networking setup you must have configured -**Minimum network requirements** +### Minimum network requirements @@ -21 +21 @@ This topic provides an overview of the networking setup you must have configured -For an optimal experience, AWS recommends reliable network connectivity of at least 100 Mbps and a maximum of 200ms round trip latency for the hybrid nodes connection to the AWS Region. The bandwidth and latency requirements can vary depending on the number of hybrid nodes and your workload characteristics such as application image size, application elasticity, monitoring and logging configurations, and application dependencies on accessing data stored in other AWS services. +For an optimal experience, we recommend that you have reliable network connectivity of at least 100 Mbps and a maximum of 200ms round trip latency for the hybrid nodes connection to the AWS Region. This is general guidance that accommodates most use cases but is not a strict requirement. The bandwidth and latency requirements can vary depending on the number of hybrid nodes and your workload characteristics, such as application image size, application elasticity, monitoring and logging configurations, and application dependencies on accessing data stored in other AWS services. We recommend that you test with your own applications and environments before deploying to production to validate that your networking setup meets the requirements for your workloads. @@ -23 +23 @@ For an optimal experience, AWS recommends reliable network connectivity of at le -**On-premises node and pod CIDRs** +### On-premises node and pod CIDRs @@ -25 +25 @@ For an optimal experience, AWS recommends reliable network connectivity of at le -Identify the node and pod CIDRs you will use for your hybrid nodes and the workloads running on them. The node CIDR is allocated from your on-premises network and the pod CIDR is allocated from your Container Network Interface (CNI) if you are using an overlay network for your CNI. You pass your on-premises node CIDRs and optionally pod CIDRs as inputs when you create your EKS cluster with the `RemoteNodeNetwork` and `RemotePodNetwork` fields. +Identify the node and pod CIDRs you will use for your hybrid nodes and the workloads running on them. The node CIDR is allocated from your on-premises network and the pod CIDR is allocated from your Container Network Interface (CNI) if you are using an overlay network for your CNI. You pass your on-premises node CIDRs and pod CIDRs as inputs when you create your EKS cluster with the `RemoteNodeNetwork` and `RemotePodNetwork` fields. Your on-premises node CIDRs must be routable on your on-premises network. See the following section for information on the on-premises pod CIDR routability. @@ -36 +36 @@ The on-premises node and pod CIDR blocks must meet the following requirements: -If your CNI performs Network Address Translation (NAT) for pod traffic as it leaves your on-premises hosts, you do not need to make your pod CIDR routable on your on-premises network or configure your EKS cluster with your _remote pod network_ for hybrid nodes to become ready to workloads. If your CNI does not use NAT for pod traffic as it leaves your on-premises hosts, your pod CIDR must be routable on your on-premises network and you must configure your EKS cluster with your remote pod network for hybrid nodes to become ready to workloads. +### On-premises pod network routing @@ -38 +38 @@ If your CNI performs Network Address Translation (NAT) for pod traffic as it lea -There are several techniques you can use to make your pod CIDR routable on your on-premises network including Border Gateway Protocol (BGP), static routes, or other custom routing solutions. BGP is the recommended solution as it is more scalable and easier to manage than alternative solutions that require custom or manual route configuration. AWS supports the BGP capabilities of Cilium and Calico for advertising hybrid nodes pod CIDRs, see [Configure CNI for hybrid nodes](./hybrid-nodes-cni.html) for more information. +When using EKS Hybrid Nodes, we generally recommend that you make your on-premises pod CIDRs routable on your on-premises network to enable full cluster communication and functionality between cloud and on-premises environments. @@ -40 +40 @@ There are several techniques you can use to make your pod CIDR routable on your -If you are running webhooks on hybrid nodes, your pod CIDR must be routable on your on-premises network and you must configure your EKS cluster with your remote pod network so the EKS control plane can directly communicate with the webhooks running on hybrid nodes. If you cannot make your pod CIDR routable on your on-premises network but need to run webhooks, it is recommended to run webhooks on cloud nodes in the same EKS cluster. For more information on running webhooks on cloud nodes, see [Configure webhooks for hybrid nodes](./hybrid-nodes-webhooks.html). +**Routable pod networks** @@ -42 +42,33 @@ If you are running webhooks on hybrid nodes, your pod CIDR must be routable on y -**Access required during hybrid node installation and upgrade** +If you are able to make your pod network routable on your on-premises network, follow the guidance below. + + 1. Configure the `RemotePodNetwork` field for your EKS cluster with your on-premises pod CIDR, your VPC route tables with your on-premises pod CIDR, and your EKS cluster security group with your on-premises pod CIDR. + + 2. There are several techniques you can use to make your on-premises pod CIDR routable on your on-premises network including Border Gateway Protocol (BGP), static routes, or other custom routing solutions. BGP is the recommended solution as it is more scalable and easier to manage than alternative solutions that require custom or manual route configuration. AWS supports the BGP capabilities of Cilium and Calico for advertising pod CIDRs, see [Configure a CNI for hybrid nodes](./hybrid-nodes-cni.html) and [Routable remote Pod CIDRs](./hybrid-nodes-concepts-kubernetes.html#hybrid-nodes-concepts-k8s-pod-cidrs) for more information. + + 3. Webhooks can run on hybrid nodes as the EKS control plane is able to communicate with the Pod IP addresses assigned to the webhooks. + + 4. Workloads running on cloud nodes are able to communicate directly with workloads running on hybrid nodes in the same EKS cluster. + + 5. Other AWS services, such as AWS Application Load Balancers and Amazon Managed Service for Prometheus, are able to communicate with workloads running on hybrid nodes to balance network traffic and scrape pod metrics. + + + + +**Unroutable pod networks** + +If you are _not_ able to make your pod networks routable on your on-premises network, follow the guidance below. + + 1. Webhooks cannot run on hybrid nodes because webhooks require connectivity from the EKS control plane to the Pod IP addresses assigned to the webhooks. In this case, we recommend that you run webhooks on cloud nodes in the same EKS cluster as your hybrid nodes, see [Configure webhooks for hybrid nodes](./hybrid-nodes-webhooks.html) for more information. + + 2. Workloads running on cloud nodes are not able to communicate directly with workloads running on hybrid nodes when using the VPC CNI for cloud nodes and Cilium or Calico for hybrid nodes. + + 3. Use Service Traffic Distribution to keep traffic local to the zone it is originating from. For more information on Service Traffic Distribution, see [Configure Service Traffic Distribution](./hybrid-nodes-webhooks.html#hybrid-nodes-mixed-service-traffic-distribution). + + 4. Configure your CNI to use egress masquerade or network address translation (NAT) for pod traffic as it leaves your on-premises hosts. This is enabled by default in Cilium. Calico requires `natOutgoing` to be set to `true`. + + 5. Other AWS services, such as AWS Application Load Balancers and Amazon Managed Service for Prometheus, are not able to communicate with workloads running on hybrid nodes. + + + + +### Access required during hybrid node installation and upgrade @@ -63 +95 @@ IAM Anywhere binary endpoint 2 | https://rolesanywhere.amazonaws.com | HTTPS | -**Access required for ongoing cluster operations** +### Access required for ongoing cluster operations @@ -88 +120 @@ User-defined | User-defined | Inbound,Outbound | App ports | Remote Pod CIDR -**Amazon EKS network interfaces** +### Amazon EKS network interfaces @@ -220 +252 @@ Run the following command for each of the subnets you created in the previous st -The following access for your EKS cluster security group is required for ongoing cluster operations. +The following access for your EKS cluster security group is required for ongoing cluster operations. Amazon EKS automatically creates the required **ingress** security group rules for hybrid nodes when you create or update your cluster with remote node and pod networks configured. @@ -229 +261,11 @@ HTTPS | TCP | Outbound | Webhook ports | N/A | Remote Pod CIDR(s) | Kubern -To create a security group with the inbound access rules, run the following commands. This security group must be passed when you create your Amazon EKS cluster. By default, the command below creates a security group that allows all outbound access. You can restrict outbound access to include only the rules above. If you’re considering limiting the outbound rules, we recommend that you thoroughly test all of your applications and pod connectivity before you apply your changed rules to a production cluster. +###### Important + +**Security group rule limits** : Amazon EC2 security groups have a maximum of 60 ingress rules by default. The security group ingress rules may not apply if your cluster security group approaches this limit. In this case, it may be required to manually add in the missing ingress rules. + +**CIDR cleanup responsibility** : If you remove remote node or pod networks from EKS clusters, EKS does not automatically remove the corresponding security group rules. You are responsible for manually removing unused remote node or pod networks from your security group rules. + +For more information about the cluster security group that Amazon EKS creates, see [View Amazon EKS security group requirements for clusters](./sec-group-reqs.html). + +### (Optional) Manual security group configuration + +If you need to create additional security groups or modify the automatically created rules, you can use the following commands as reference. By default, the command below creates a security group that allows all outbound access. You can restrict outbound access to include only the rules above. If you’re considering limiting the outbound rules, we recommend that you thoroughly test all of your applications and pod connectivity before you apply your changed rules to a production cluster.