AWS eks documentation change
Summary
Removed Calico support and documentation, updated Cilium as the sole AWS-supported CNI for hybrid nodes. Added detailed Cilium configuration guidance, version updates (v1.17.x), and moved Calico examples to external repository.
Security assessment
Changes focus on CNI support lifecycle and configuration best practices rather than addressing specific vulnerabilities. Security documentation additions include explicit guidance about pod CIDR routability requirements when disabling masquerading, and recommendations for Cilium BGP configuration to ensure proper network isolation.
Diff
diff --git a/eks/latest/userguide/hybrid-nodes-cni.md b/eks/latest/userguide/hybrid-nodes-cni.md index ef4b3053b..52b11a472 100644 --- a//eks/latest/userguide/hybrid-nodes-cni.md +++ b//eks/latest/userguide/hybrid-nodes-cni.md @@ -5 +5 @@ -CNI version compatibilitySupported capabilitiesCilium considerationsInstall Cilium on hybrid nodesUpgrade Cilium on hybrid nodesDelete Cilium from hybrid nodesCalico considerationsInstall Calico on hybrid nodesUpgrade Calico on hybrid nodesDelete Calico from hybrid nodes +Version compatibilitySupported capabilitiesCilium considerationsInstall Cilium on hybrid nodesUpgrade Cilium on hybrid nodesDelete Cilium from hybrid nodes @@ -11 +11 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -# Configure a CNI for hybrid nodes +# Configure CNI for hybrid nodes @@ -13 +13 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -Cilium and Calico are supported as the Container Networking Interfaces (CNIs) for Amazon EKS Hybrid Nodes. You must install a CNI for hybrid nodes to become ready to serve workloads. Hybrid nodes appear with status `Not Ready` until a CNI is running. You can manage these CNIs with your choice of tools such as Helm. The Amazon VPC CNI is not compatible with hybrid nodes and the VPC CNI is configured with anti-affinity for the `eks.amazonaws.com/compute-type: hybrid` label. +Cilium is the AWS-supported Container Networking Interface (CNI) for Amazon EKS Hybrid Nodes. You must install a CNI for hybrid nodes to become ready to serve workloads. Hybrid nodes appear with status `Not Ready` until a CNI is running. You can manage the CNI with your choice of tools such as Helm. The instructions on this page cover Cilium lifecycle management (install, upgrade, delete). See [Cilium Ingress and Cilium Gateway Overview](./hybrid-nodes-ingress.html#hybrid-nodes-ingress-cilium), [Service type LoadBalancer](./hybrid-nodes-ingress.html#hybrid-nodes-ingress-cilium-loadbalancer), and [Configure Kubernetes Network Policies for hybrid nodes](./hybrid-nodes-network-policies.html) for how to configure Cilium for ingress, load balancing, and network policies. @@ -15 +15 @@ Cilium and Calico are supported as the Container Networking Interfaces (CNIs) fo -## CNI version compatibility +Cilium is not supported by AWS when running on nodes in AWS Cloud. The Amazon VPC CNI is not compatible with hybrid nodes and the VPC CNI is configured with anti-affinity for the `eks.amazonaws.com/compute-type: hybrid` label. @@ -17 +17 @@ Cilium and Calico are supported as the Container Networking Interfaces (CNIs) fo -Cilium version `v1.16.x` is supported and recommended for EKS Hybrid Nodes for every Kubernetes version supported in Amazon EKS. +The Calico documentation previously on this page has been moved to the [EKS Hybrid Examples Repository](https://github.com/aws-samples/eks-hybrid-examples). @@ -19 +19,3 @@ Cilium version `v1.16.x` is supported and recommended for EKS Hybrid Nodes for e -Calico version `v3.29.x` is supported and recommended for EKS Hybrid Nodes for every Kubernetes version supported in Amazon EKS. +## Version compatibility + +Cilium version `v1.17.x` is supported for EKS Hybrid Nodes for every Kubernetes version supported in Amazon EKS. @@ -25,12 +27,18 @@ See [Kubernetes version support](https://docs.aws.amazon.com/eks/latest/userguid -AWS provides technical support for the following capabilities of Cilium and Calico for use with hybrid nodes. If you plan to use functionality outside the scope of AWS support, we recommend that you obtain commercial support for the plugin or have the in-house expertise to troubleshoot and contribute fixes to the CNI plugin project. - -Feature | Cilium | Calico ----|---|--- -Kubernetes network conformance | Yes | Yes -Control plane to node connectivity | Yes | Yes -Control plane to pod connectivity | Yes | Yes -Lifecycle Management | Install, Upgrade, Delete | Install, Upgrade, Delete -Networking Mode | VXLAN | VXLAN -IP Address Management (IPAM) | Cluster Scope (Cilium IPAM) | Calico IPAM -IP family | IPv4 | IPv4 -BGP | Yes (Cilium Control Plane) | Yes +AWS maintains builds of Cilium for EKS Hybrid Nodes that are based on the open source [Cilium project](https://github.com/cilium/cilium). To receive support from AWS for Cilium, you must be using the AWS-maintained Cilium builds and supported Cilium versions. + +AWS provides technical support for the default configurations of the following capabilities of Cilium for use with EKS Hybrid Nodes. If you plan to use functionality outside the scope of AWS support, it is recommended to obtain alternative commercial support for Cilium or have the in-house expertise to troubleshoot and contribute fixes to the Cilium project. + +Cilium Feature | Supported by AWS +---|--- +Kubernetes network conformance | Yes +Core cluster connectivity | Yes +IP family | IPv4 +Lifecycle Management | Helm +Networking Mode | VXLAN encapsulation +IP Address Management (IPAM) | Cilium IPAM Cluster Scope +Network Policy | Kubernetes Network Policy +Border Gateway Protocol (BGP) | Cilium BGP Control Plane +Kubernetes Ingress | Cilium Ingress, Cilium Gateway +Service LoadBalancer IP Allocation | Cilium Load Balancer IPAM +Service LoadBalancer IP Address Advertisement | Cilium BGP Control Plane +kube-proxy replacement | Yes @@ -39,0 +48,2 @@ BGP | Yes (Cilium Control Plane) | Yes + * **Helm repository** \- AWS hosts the Cilium Helm chart in the Amazon Elastic Container Registry Public (Amazon ECR Public) at [Amazon EKS Cilium/Cilium](https://gallery.ecr.aws/eks/cilium/cilium). You can use the following URI in your `helm` commands to use this repository: `oci://public.ecr.aws/eks/cilium/cilium:1.17.6-0`. The commands in this topic use this repository. Note that certain `helm repo` commands aren’t valid for Helm repositores in Amazon ECR Public, so you can’t refer to this repository from a local Helm repo name. Instead, use the full URI in most commands. + @@ -42 +52 @@ BGP | Yes (Cilium Control Plane) | Yes - * By default, Cilium [masquerades](https://docs.cilium.io/en/stable/network/concepts/masquerading/) the source IP address of all pod traffic leaving the cluster to the IP address of the node. This makes it possible to run Cilium with hybrid nodes, whether or not remote pod networks are configured on the cluster. If you disable masquerading, then your pod CIDRs must be routable on your on-premises network and you must configure your Amazon EKS cluster with your remote pod networks. + * By default, Cilium [masquerades](https://docs.cilium.io/en/stable/network/concepts/masquerading/) the source IP address of all pod traffic leaving the cluster to the IP address of the node. If you disable masquerading, then your pod CIDRs must be routable on your on-premises network. @@ -44 +54 @@ BGP | Yes (Cilium Control Plane) | Yes - * If you are running webhooks on your hybrid nodes, your pod CIDRs must be routable on your on-premises network and you must configure your Amazon EKS cluster with your remote pod networks. If your pod CIDRs are not routable on your on-premises network, then we recommend that you run webhooks on cloud nodes in the same cluster. See [Configure webhooks for hybrid nodes](./hybrid-nodes-webhooks.html) for more information. + * If you are running webhooks on hybrid nodes, your pod CIDRs must be routable on your on-premises network. If your pod CIDRs are not routable on your on-premises network, then it is recommended to run webhooks on cloud nodes in the same cluster. See [Configure webhooks for hybrid nodes](./hybrid-nodes-webhooks.html) and [Prepare networking for hybrid nodes](./hybrid-nodes-networking.html) for more information. @@ -46 +56 @@ BGP | Yes (Cilium Control Plane) | Yes - * A common way to make your pod CIDR routable on your on-premises network is to advertise pod addresses with BGP. To use BGP with Cilium, you must set `bgpControlPlane.enabled: true` in your Helm configuration. For more information on Cilium’s BGP support, see [Cilium BGP Control Plane](https://docs.cilium.io/en/stable/network/bgp-control-plane/bgp-control-plane/) in the Cilium documentation. + * AWS recommends using Cilium’s built-in BGP functionality to make your pod CIDRs routable on your on-premises network. For more information on how to configure Cilium BGP with hybrid nodes, see [Configure Cilium BGP for hybrid nodes](./hybrid-nodes-cilium-bgp.html). @@ -48 +58 @@ BGP | Yes (Cilium Control Plane) | Yes - * The default IP Address Management (IPAM) in Cilium is called [Cluster Scope](https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/), where the Cilium operator allocates IP addresses for each node based on user-configured pod CIDRs. The pod CIDRs are configured with the `clusterPoolIPv4PodCIDRList` Helm value, which should match the remote pod network CIDRs you configured for your Amazon EKS cluster. Cilium allocates segments from the `clusterPoolIPv4PodCIDRList` to each node. The size of the per node segments is configured with the `clusterPoolIPv4MaskSize` Helm value. For more information on the `clusterPoolIPv4PodCIDRList` and `clusterPoolIPv4MaskSize`, see [Expanding the cluster pool](https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/#expanding-the-cluster-pool) in the Cilium documentation. + * The default IP Address Management (IPAM) in Cilium is called [Cluster Scope](https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/), where the Cilium operator allocates IP addresses for each node based on user-configured pod CIDRs. @@ -55 +65 @@ BGP | Yes (Cilium Control Plane) | Yes - 1. Ensure that you have installed the Helm CLI on your command-line environment. See the [setup helm](./helm.html) for installation instructions. +### Procedure @@ -57 +67 @@ BGP | Yes (Cilium Control Plane) | Yes - 2. Install the Cilium Helm repo. + 1. Create a YAML file called `cilium-values.yaml`. The following example configures Cilium to run only on hybrid nodes by setting affinity for the `eks.amazonaws.com/compute-type: hybrid` label for the Cilium agent and operator. @@ -59 +69 @@ BGP | Yes (Cilium Control Plane) | Yes - helm repo add cilium https://helm.cilium.io/ + * Configure `clusterPoolIpv4PodCIDRList` with the same pod CIDRs you configured for your EKS cluster’s _remote pod networks_. For example, `10.100.0.0/24`. The Cilium operator allocates IP address slices from within the configured `clusterPoolIpv4PodCIDRList` IP space. Your pod CIDR must not overlap with your on-premises node CIDR, your VPC CIDR, or your Kubernetes service CIDR. @@ -61 +71 @@ BGP | Yes (Cilium Control Plane) | Yes - 3. Create a YAML file called `cilium-values.yaml`. The following example configures Cilium to run only on hybrid nodes by setting affinity for the `eks.amazonaws.com/compute-type: hybrid` label. + * Configure `clusterPoolIpv4MaskSize` based on your required pods per node. For example, `25` for a /25 segment size of 128 pods per node. @@ -63 +73 @@ BGP | Yes (Cilium Control Plane) | Yes - * If you configured your Amazon EKS cluster with _remote pod networks_ , configure the same pod CIDRs for your `clusterPoolIPv4PodCIDRList`. For example, `10.100.0.0/24`. Your on-premises pod CIDR must not overlap with your on-premises node CIDR when running the CNI in overlay / tunnel mode. + * Do not change `clusterPoolIpv4PodCIDRList` or `clusterPoolIpv4MaskSize` after deploying Cilium on your cluster, see [Expanding the cluster pool](https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/#expanding-the-cluster-pool) for more information. @@ -65 +75 @@ BGP | Yes (Cilium Control Plane) | Yes - * Configure `clusterPoolIPv4MaskSize` based on your required pods per node. For example, `25` for a /25 segment size of 128 pods per node. + * If you are running Cilium in kube-proxy replacement mode, set `kubeProxyReplacement: "true"` in your Helm values and ensure you do not have an existing kube-proxy deployment running on the same nodes as Cilium. @@ -67 +77,3 @@ BGP | Yes (Cilium Control Plane) | Yes - * You should not change your `clusterPoolIPv4PodCIDRList` or `clusterPoolIPv4MaskSize` after deploying Cilium on your cluster, see [Expanding the cluster pool](https://docs.cilium.io/en/stable/network/concepts/ipam/cluster-pool/#expanding-the-cluster-pool) in the Cilium documentation. + * The example below disables the Envoy Layer 7 (L7) proxy that Cilium uses for L7 network policies and ingress. For more information, see [Configure Kubernetes Network Policies for hybrid nodes](./hybrid-nodes-network-policies.html) and [Cilium Ingress and Cilium Gateway Overview](./hybrid-nodes-ingress.html#hybrid-nodes-ingress-cilium). + + * The example below configures `loadBalancer.serviceTopology`: `true` for Service Traffic Distribution to function correctly if you configure it for your services. For more information, see [Configure Service Traffic Distribution](./hybrid-nodes-webhooks.html#hybrid-nodes-mixed-service-traffic-distribution). @@ -86,2 +97,0 @@ BGP | Yes (Cilium Control Plane) | Yes - loadBalancer: - serviceTopology: true @@ -99,0 +110,2 @@ BGP | Yes (Cilium Control Plane) | Yes + loadBalancer: + serviceTopology: true @@ -101,0 +114 @@ BGP | Yes (Cilium Control Plane) | Yes + kubeProxyReplacement: "false" @@ -103,3 +116 @@ BGP | Yes (Cilium Control Plane) | Yes - 4. Install Cilium on your cluster. - - * Replace `CILIUM_VERSION` with a Cilium version (for example `v1.17.0`). We recommend that you run the latest patch version for your Cilium minor version. You can find the latest patch release for a given minor Cilium release in the [Stable Releases section](https://github.com/cilium/cilium#stable-releases) of the Cilium documentation. + 2. Install Cilium on your cluster. @@ -107 +118 @@ BGP | Yes (Cilium Control Plane) | Yes - * If you are enabling BGP for your deployment, add the `--set bgpControlPlane.enabled=true` flag in the command below. + * Replace `CILIUM_VERSION` with a Cilium version (for example `1.17.5`). It is recommended to use the latest patch version for the Cilium minor version. You can find the latest patch release available in your local Helm repository with the `helm search repo cilium/cilium --versions` command. @@ -111 +122 @@ BGP | Yes (Cilium Control Plane) | Yes - helm install cilium cilium/cilium \ + helm install cilium oci://public.ecr.aws/eks/cilium/cilium \ @@ -116 +127 @@ BGP | Yes (Cilium Control Plane) | Yes - 5. You can confirm your Cilium installation was successful with the following commands. You should see the `cilium-operator` deployment and the `cilium-agent` running on each of your hybrid nodes. Additionally, your hybrid nodes should now have status `Ready`. For information on how to configure BGP for Cilium, proceed to the next step. + 3. Confirm your Cilium installation was successful with the following commands. You should see the `cilium-operator` deployment and the `cilium-agent` running on each of your hybrid nodes. Additionally, your hybrid nodes should now have status `Ready`. For information on how to configure Cilium BGP to advertise your pod CIDRs to your on-premises network, proceed to [Configure Cilium BGP for hybrid nodes](./hybrid-nodes-cilium-bgp.html). @@ -129,83 +139,0 @@ BGP | Yes (Cilium Control Plane) | Yes - 6. To use BGP with Cilium to advertise your pod addresses with your on-premises network, you must have installed Cilium with `bgpControlPlane.enabled: true`. If you are enabling BGP for an existing Cilium deployment, you must restart the Cilium operator to apply the BGP configuration if BGP was not previously enabled. You can set `operator.rollOutPods` to `true` in your Helm values to restart the Cilium operator as part of the Helm install/upgrade process. - -To configure BGP in Cilium, first create a file called `cilium-bgp-cluster.yaml` with a `CiliumBGPClusterConfig` definition. You may need to obtain the following information from your network administrator. - - * Configure `localASN` with the ASN for the nodes running Cilium. - - * Configure `peerASN` with the ASN for your on-premises router. - - * Configure the `peerAddress` with the on-premises router IP that each node running Cilium will peer with. - - apiVersion: cilium.io/v2alpha1 - kind: CiliumBGPClusterConfig - metadata: - name: cilium-bgp - spec: - nodeSelector: - matchExpressions: - - key: eks.amazonaws.com/compute-type - operator: In - values: - - hybrid - bgpInstances: - - name: "rack0" - localASN: NODES_ASN - peers: - - name: "onprem-router" - peerASN: ONPREM_ROUTER_ASN - peerAddress: ONPREM_ROUTER_IP - peerConfigRef: - name: "cilium-peer" - - 7. Apply the Cilium BGP Cluster configuration to your cluster. - - kubectl apply -f cilium-bgp-cluster.yaml - - 8. The `CiliumBGPPeerConfig` resource defines a BGP peer configuration. Multiple peers can share the same configuration and provide reference to the common `CiliumBGPPeerConfig` resource. Create a file named `cilium-bgp-peer.yaml` to configure the peer configuration for your on-premises network. See the [BGP Peer Configuration](https://docs.cilium.io/en/latest/network/bgp-control-plane/bgp-control-plane-v2/#bgp-peer-configuration) in the Cilium documentation for a full list of configuration options. - - apiVersion: cilium.io/v2alpha1 - kind: CiliumBGPPeerConfig - metadata: - name: cilium-peer - spec: - timers: - holdTimeSeconds: 30 - keepAliveTimeSeconds: 10 - gracefulRestart: - enabled: true - restartTimeSeconds: 120 - families: - - afi: ipv4 - safi: unicast - advertisements: - matchLabels: - advertise: "bgp" - - 9. Apply the Cilium BGP Peer configuration to your cluster. - - kubectl apply -f cilium-bgp-peer.yaml - - 10. The `CiliumBGPAdvertisement` resource is used to define various advertisement types and attributes associated with them. Create a file named `cilium-bgp-advertisement.yaml` and configure the `CiliumBGPAdvertisement` resource with your settings. - - apiVersion: cilium.io/v2alpha1 - kind: CiliumBGPAdvertisement - metadata: - name: bgp-advertisements - labels: - advertise: bgp - spec: - advertisements: - - advertisementType: "PodCIDR" - - advertisementType: "Service" - service: - addresses: - - ClusterIP - - ExternalIP - - LoadBalancerIP - - 11. Apply the Cilium BGP Advertisement configuration to your cluster. - - kubectl apply -f cilium-bgp-advertisement.yaml - -You can confirm the BGP peering worked with the [Cilium CLI](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli) by using the `cilium bgp peers` command. You should see the correct values in the output for your environment and the Session State as `established`. See the [Troubleshooting and Operations Guide](https://docs.cilium.io/en/latest/network/bgp-control-plane/bgp-control-plane/#troubleshooting-and-operation-guide) in the Cilium documentation for more information on troubleshooting. - @@ -217 +145 @@ You can confirm the BGP peering worked with the [Cilium CLI](https://docs.cilium