AWS eks documentation change
Summary
Updated documentation for webhook configuration in hybrid EKS clusters, including Service Traffic Distribution guidance, cert-manager integration, and clarifications for running webhooks in AWS Cloud
Security assessment
The changes add detailed configuration guidance for cert-manager (a security-critical component handling TLS certificates) to run on AWS Cloud nodes, which improves security posture by ensuring certificate management occurs in a controlled environment. However, there is no evidence of addressing a specific disclosed vulnerability.
Diff
diff --git a/eks/latest/userguide/hybrid-nodes-webhooks.md b/eks/latest/userguide/hybrid-nodes-webhooks.md index 35b160506..990cfc0c4 100644 --- a//eks/latest/userguide/hybrid-nodes-webhooks.md +++ b//eks/latest/userguide/hybrid-nodes-webhooks.md @@ -13 +13 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -This page details considerations for running webhooks with hybrid nodes. Webhooks are used in Kubernetes applications and open-source projects, such as the AWS Load Balancer Controller and CloudWatch Observability Agent, to perform mutating and validation capabilities at runtime. +This page details considerations for running webhooks with hybrid nodes. Webhooks are used in Kubernetes applications and open source projects, such as the AWS Load Balancer Controller and CloudWatch Observability Agent, to perform mutating and validation capabilities at runtime. @@ -15 +15 @@ This page details considerations for running webhooks with hybrid nodes. Webhook -If you are running webhooks on hybrid nodes, your on-premises pod CIDR must be routable on your on-premises network and you must configure your EKS cluster with your remote pod network so the EKS control plane can communicate with the webhooks running on hybrid nodes. +If you are running webhooks on hybrid nodes, your on-premises pod CIDR must be routable on your on-premises network. Also you must configure your EKS cluster with your remote pod network so the EKS control plane can communicate with the webhooks running on hybrid nodes. @@ -19 +19 @@ There are several techniques you can use to make your on-premises pod CIDR routa -If you _cannot_ make your on-premises pod CIDR routable on your on-premises network and need to run webhooks, it is recommended to run webhooks on EC2 instances in the same EKS cluster as your hybrid nodes. +If you _cannot_ make your on-premises pod CIDR routable on your on-premises network and need to run webhooks, we recommend that you run all of your webhooks in the AWS Cloud. To function, a webhook must run in the same EKS cluster as your hybrid nodes. @@ -23 +23 @@ If you _cannot_ make your on-premises pod CIDR routable on your on-premises netw -Mixed mode clusters are defined as EKS clusters that have both hybrid nodes and nodes running in AWS Cloud. When running a mixed mode cluster, consider the following recommendations: +_Mixed mode clusters_ are defined as EKS clusters that have both hybrid nodes and nodes running in AWS Cloud. When running a mixed mode cluster, consider the following recommendations: @@ -37 +37 @@ Mixed mode clusters are defined as EKS clusters that have both hybrid nodes and - * To collect metrics for hybrid nodes using Amazon Managed Service for Prometheus (AMP) managed collectors, your on-premises pod CIDR must be routable on your on-premises network. You can alternatively use the AMP managed collector for EKS control plane metrics and nodes running in AWS Cloud, and the AWS Distro for OpenTelemetry (ADOT) add-on to collect metrics for hybrid nodes. + * To collect metrics for hybrid nodes using Amazon Managed Service for Prometheus (AMP) managed collectors, your on-premises pod CIDR must be routable on your on-premises network. Or, you can use the AMP managed collector for EKS control plane metrics and nodes running in AWS Cloud, and the AWS Distro for OpenTelemetry (ADOT) add-on to collect metrics for hybrid nodes. @@ -54 +54 @@ To view the mutating and validating webhooks running on your cluster, you can vi -If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS Cloud, it is recommended to have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in AWS Cloud. The CoreDNS Service can be configured to prefer the closest CoreDNS replica to prevent latency and network issues in a mixed mode cluster setup with the following steps. +If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS Cloud, we recommend that you have at least one CoreDNS replica on hybrid nodes and at least one CoreDNS replica on your nodes in AWS Cloud. To prevent latency and network issues in a mixed mode cluster setup, you can configure the CoreDNS Service to prefer the closest CoreDNS replica with [Service Traffic Distribution](https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-distribution). @@ -56 +56,5 @@ If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS - 1. Add a topology zone label for each of your hybrid nodes, for example `topology.kubernetes.io/zone: onprem`. This can alternatively be done at the `nodeadm init` phase by specifying the label in your `nodeadm` configuration. Note, nodes running in AWS Cloud automatically get a topology zone label applied to them that corresponds to the availability zone (AZ) of the node. +_Service Traffic Distribution_ (available for Kubernetes versions 1.31 and later in EKS) is the recommended solution over [Topology Aware Routing](https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/) because it is more predictable. In Service Traffic Distribution, healthy endpoints in the zone will receive all of the traffic for that zone. In Topology Aware Routing, each service must meet several conditions in that zone to apply the custom routing, otherwise it routes traffic evenly to all endpoints. The following steps configure Service Traffic Distribution. + +If you are using Cilium as your CNI, you must run the CNI with the `enable-service-topology` set to `true` to enable Service Traffic Distribution. You can pass this configuration with the Helm install flag `--set loadBalancer.serviceTopology=true` or you can update an existing installation with the Cilium CLI command `cilium config set enable-service-topology true`. The Cilium agent running on each node must be restarted after updating the configuration for an existing installation. + + 1. Add a topology zone label for each of your hybrid nodes, for example `topology.kubernetes.io/zone: onprem`. Or, you can set the label at the `nodeadm init` phase by specifying the label in your `nodeadm` configuration, see [Node Config for customizing kubelet (Optional)](./hybrid-nodes-nodeadm.html#hybrid-nodes-nodeadm-kubelet). Note, nodes running in AWS Cloud automatically get a topology zone label applied to them that corresponds to the availability zone (AZ) of the node. @@ -61 +65 @@ If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS - 2. Add `podAntiAffinity` to the CoreDNS deployment configuration with the topology zone key. You can alternatively configure the CoreDNS deployment during installation with EKS add-ons. + 2. Add `podAntiAffinity` to the CoreDNS deployment with the topology zone key. Or, you can configure the CoreDNS deployment during installation with EKS add-ons. @@ -92 +96 @@ If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS - 3. Add `trafficDistribution` to the kube-dns Service configuration. + 3. Add the setting `trafficDistribution: PreferClose` to the `kube-dns` Service configuration to enable Topology Aware Routing. @@ -94 +98,5 @@ If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS - kubectl edit service kube-dns -n kube-system + kubectl patch svc kube-dns -n kube-system --type=merge -p '{ + "spec": { + "trafficDistribution": "PreferClose" + } + }' @@ -96,3 +104,23 @@ If you are running a mixed mode cluster with both hybrid nodes and nodes in AWS - spec: - ... - trafficDistribution: PreferClose + 4. You can confirm that Service Traffic Distribution is enabled by viewing the endpoint slices for the `kube-dns` Service. Your endpoint slices must show the `hints` for your topology zone labels, which confirms that Service Traffic Distribution is enabled. If you do not see the `hints` for each endpoint address, then Service Traffic Distribution is not enabled. + + kubectl get endpointslice -A | grep "kube-dns" + + kubectl get endpointslice [.replaceable]`kube-dns-<id>` -n kube-system -o yaml + + addressType: IPv4 + apiVersion: discovery.k8s.io/v1 + endpoints: + - addresses: + - <your-hybrid-node-pod-ip> + hints: + forZones: + - name: onprem + nodeName: <your-hybrid-node-name> + zone: onprem + - addresses: + - <your-cloud-node-pod-ip> + hints: + forZones: + - name: us-west-2a + nodeName: <your-cloud-node-name> + zone: us-west-2a @@ -112,0 +141,2 @@ The following add-ons use webhooks and are supported for use with hybrid nodes. + * `cert-manager` + @@ -116 +146 @@ The following add-ons use webhooks and are supported for use with hybrid nodes. -See the sections below for configuring the webhooks used by these add-ons to run on nodes in AWS Cloud. +See the following sections for configuring the webhooks used by these add-ons to run on nodes in AWS Cloud. @@ -120 +150 @@ See the sections below for configuring the webhooks used by these add-ons to run -To run the AWS Load Balancer Controller on nodes in AWS Cloud in a mixed mode cluster setup, add the following to your Helm values configuration or specify the values using EKS add-on configuration. +To use the AWS Load Balancer Controller in a mixed mode cluster setup, you must run the controller on nodes in AWS Cloud. To do so, add the following to your Helm values configuration or specify the values by using EKS add-on configuration. @@ -135 +165 @@ To run the AWS Load Balancer Controller on nodes in AWS Cloud in a mixed mode cl -The CloudWatch Observability Agent add-on has an operator that uses webhooks. To run the operator on nodes in AWS Cloud in a mixed mode cluster setup, edit the CloudWatch Observability Agent operator configuration. The ability to configure operator affinity during installation with Helm and EKS add-ons is planned for a future release (see [containers-roadmap issue #2431](https://github.com/aws/containers-roadmap/issues/2431)). +The CloudWatch Observability Agent add-on has a Kubernetes Operator that uses webhooks. To run the operator on nodes in AWS Cloud in a mixed mode cluster setup, edit the CloudWatch Observability Agent operator configuration. You can’t configure the operator affinity during installation with Helm and EKS add-ons (see [containers-roadmap issue #2431](https://github.com/aws/containers-roadmap/issues/2431)). @@ -158 +188 @@ The CloudWatch Observability Agent add-on has an operator that uses webhooks. To -The AWS Distro for OpenTelemetry (ADOT) add-on has an operator that uses webhooks. To run the operator on nodes in AWS Cloud in a mixed mode cluster setup, add the following to your Helm values configuration or specify the values using EKS add-on configuration. +The AWS Distro for OpenTelemetry (ADOT) add-on has a Kubernetes Operator that uses webhooks. To run the operator on nodes in AWS Cloud in a mixed mode cluster setup, add the following to your Helm values configuration or specify the values by using EKS add-on configuration. @@ -171 +201 @@ The AWS Distro for OpenTelemetry (ADOT) add-on has an operator that uses webhook -If your pod CIDR is not routable on your on-premises network, configure the ADOT collector Custom Resource Definition (CRD) to run on your hybrid nodes so it can scrape the metrics from your hybrid nodes and the workloads running on them. +If your pod CIDR is not routable on your on-premises network, then the ADOT collector must run on hybrid nodes to scrape the metrics from your hybrid nodes and the workloads running on them. To do so, edit the Custom Resource Definition (CRD). @@ -197 +227,46 @@ You can configure the ADOT collector to only scrape metrics from hybrid nodes an -The ADOT add-on has a prerequisite requirement to install cert-manager for the TLS certificates used by the ADOT operator webhook. Cert-manager also runs webhooks and you can configure it to run on nodes in AWS Cloud with the following Helm values configuration. +The ADOT add-on has a prerequisite requirement to install `cert-manager` for the TLS certificates used by the ADOT operator webhook. `cert-manager` also runs webhooks and you can configure it to run on nodes in AWS Cloud with the following Helm values configuration. + + + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - hybrid + webhook: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - hybrid + cainjector: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - hybrid + startupapicheck: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: eks.amazonaws.com/compute-type + operator: NotIn + values: + - hybrid + +#### `cert-manager` + +The `cert-manager` add-on runs webhooks and you can configure it to run on nodes in AWS Cloud with the following Helm values configuration.