AWS eks documentation change
Summary
Restructured data plane documentation to emphasize EKS Auto Mode as primary scaling solution, removed detailed Cluster Autoscaler configuration guidance, added Karpenter references, simplified EBS volume/AZ guidance, and updated CoreDNS management recommendations
Security assessment
Changes focus on operational best practices and feature updates (Auto Mode/Karpenter) rather than addressing security vulnerabilities. No explicit security vulnerabilities or security feature additions are mentioned in the diff
Diff
diff --git a/eks/latest/best-practices/data-plane.md b/eks/latest/best-practices/data-plane.md index 9fac354ef..a9a083ca7 100644 --- a//eks/latest/best-practices/data-plane.md +++ b//eks/latest/best-practices/data-plane.md @@ -5 +5 @@ -RecommendationsCoreDNSRecommendations +Recommendations @@ -11 +11 @@ To operate high-available and resilient applications, you need a highly-availabl -You have two choices for worker nodes with EKS: [EC2 instances](https://docs.aws.amazon.com/eks/latest/userguide/worker.html) and [Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html). If you choose EC2 instances, you can manage the worker nodes yourself or use [EKS managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html). You can have a cluster with a mix of managed, self-managed worker nodes, and Fargate. +You have multiple choices for worker nodes with EKS: [EKS Auto Mode managed nodes](https://docs.aws.amazon.com/eks/latest/userguide/automode.html), [EC2 Instances](https://docs.aws.amazon.com/eks/latest/userguide/worker.html) and [Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate.html). @@ -13 +13 @@ You have two choices for worker nodes with EKS: [EC2 instances](https://docs.aws -EKS on Fargate offers the easiest path to a resilient data plane. Fargate runs each Pod in an isolated compute environment. Each Pod running on Fargate gets its own worker node. Fargate automatically scales the data plane as Kubernetes scales pods. You can scale both the data plane and your workload by using the [horizontal pod autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html). +EKS Auto Mode offers the easiest path to a resilient data plane. Auto Mode extends AWS management of Kubernetes clusters beyond the cluster itself, to allow AWS to also set up and manage the infrastructure that enables the smooth operation of your workloads. Auto Mode automatically scales the data plane up or down as Kubernetes scales Pods and works to continually ensure that the Nodes in your cluster are sized appropriately and cost-effectively for the currently running workloads. @@ -15 +15 @@ EKS on Fargate offers the easiest path to a resilient data plane. Fargate runs e -The preferred way to scale EC2 worker nodes is by using [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), [EC2 Auto Scaling groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html) or community projects like [Atlassian’s Escalator](https://github.com/atlassian/escalator). +If you choose EC2 instances, you can manage the worker nodes yourself or use [EKS managed node groups](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html). You can have a cluster with a mix of Auto Mode, managed, self-managed worker nodes, and Fargate. @@ -17,17 +17 @@ The preferred way to scale EC2 worker nodes is by using [Kubernetes Cluster Auto -## Recommendations - -### Use EC2 Auto Scaling Groups to create worker nodes - -It is a best practice to create worker nodes using EC2 Auto Scaling groups instead of creating individual EC2 instances and joining them to the cluster. Auto Scaling Groups will automatically replace any terminated or failed nodes ensuring that the cluster always has the capacity to run your workload. - -### Use Kubernetes Cluster Autoscaler to scale nodes - -Cluster Autoscaler adjusts the size of the data plane when there are pods that cannot be run because the cluster has insufficient resources, and adding another worker node would help. Although Cluster Autoscaler is a reactive process, it waits until pods are in _Pending_ state due to insufficient capacity in the cluster. When such an event occurs, it adds EC2 instances to the cluster. Whenever the cluster runs out of capacity, new replicas - or new pods - will be unavailable (_in Pending state_) until worker nodes are added. This delay may impact your applications’ reliability if the data plane cannot scale fast enough to meet the demands of the workload. If a worker node is consistently underutilized and all of its pods can be scheduled on other worker nodes, Cluster Autoscaler terminates it. - -### Configure over-provisioning with Cluster Autoscaler - -Cluster Autoscaler triggers a scale-up of the data-plane when Pods in the cluster are already _Pending_. Hence, there may be a delay between the time your application needs more replicas, and when it, in fact, gets more replicas. An option to account for this possible delay is through adding more than required replicas, inflating the number of replicas for the application. - -Another pattern that Cluster Autoscaler recommends uses [_pause_ Pods and the Priority Preemption feature](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler). The _pause Pod_ runs a [pause container](https://github.com/kubernetes/kubernetes/tree/master/build/pause), which as the name suggests, does nothing but acts as a placeholder for compute capacity that can be used by other Pods in your cluster. Because it runs with a _very low assigned priority_ , the pause Pod gets evicted from the node when another Pod needs to be created, and the cluster doesn’t have available capacity. The Kubernetes Scheduler notices the eviction of the pause Pod and tries to reschedule it. But since the cluster is running at capacity, the pause Pod remains _Pending_ , to which the Cluster Autoscaler reacts by adding nodes. - -### Using Cluster Autoscaler with multiple Auto Scaling Groups +Fargate runs each Pod in an isolated compute environment. Each Pod running on Fargate gets its own worker node. Fargate automatically scales the data plane as Kubernetes scales pods. You can scale both the data plane and your workload by using the [horizontal pod autoscaler](https://docs.aws.amazon.com/eks/latest/userguide/horizontal-pod-autoscaler.html). @@ -35 +19 @@ Another pattern that Cluster Autoscaler recommends uses [_pause_ Pods and the Pr -Run the Cluster Autoscaler with the `--node-group-auto-discovery` flag enabled. Doing so will allow the Cluster Autoscaler to find all autoscaling groups that include a particular defined tag and prevents the need to define and maintain each autoscaling group in the manifest. +The preferred way to scale EC2 worker nodes (if not using EKS Auto Mode where this is performed automatically by AWS) is by using [Karpenter](https://karpenter.sh/), [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), or [EC2 Auto Scaling groups](https://docs.aws.amazon.com/autoscaling/ec2/userguide/AutoScalingGroup.html). @@ -37,3 +21 @@ Run the Cluster Autoscaler with the `--node-group-auto-discovery` flag enabled. -### Using Cluster Autoscaler with local storage - -By default, the Cluster Autoscaler does not scale-down nodes that have pods deployed with local storage attached. Set the `--skip-nodes-with-local-storage` flag to false to allow Cluster Autoscaler to scale-down these nodes. +## Recommendations @@ -41 +23 @@ By default, the Cluster Autoscaler does not scale-down nodes that have pods depl -### Spread worker nodes and workload across multiple AZs +### Spread worker nodes and workloads across multiple AZs @@ -43 +25 @@ By default, the Cluster Autoscaler does not scale-down nodes that have pods depl -You can protect your workloads from failures in an individual AZ by running worker nodes and pods in multiple AZs. You can control the AZ the worker nodes are created in using the subnets you create the nodes in. +You can protect your workloads from failures in an individual AZ by running worker nodes and Pods in multiple AZs. You can control the AZ the worker nodes are created in using the subnets you create the nodes in. @@ -45 +27 @@ You can protect your workloads from failures in an individual AZ by running work -If you are using Kubernetes 1.18+, the recommended method for spreading pods across AZs is to use [Topology Spread Constraints for Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#spread-constraints-for-pods). +The recommended method for spreading pods across AZs is to use [Topology Spread Constraints for Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#spread-constraints-for-pods). Auto-scaling capabilities like EKS Auto Mode and Karpenter are aware of topology spread constraints and will automatically launch Nodes in the correct AZs to allow your constraints to be met. @@ -80 +62 @@ The deployment below spreads pods across AZs if possible, letting those pods run -`kube-scheduler` is only aware of topology domains via nodes that exist with those labels. If the above deployment is deployed to a cluster with nodes only in a single zone, all of the pods will schedule on those nodes as `kube-scheduler` isn’t aware of the other zones. For this topology spread to work as expected with the scheduler, nodes must already exist in all zones. This issue will be resolved in Kubernetes 1.24 with the addition of the `MinDomainsInPodToplogySpread` [feature gate](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#api) which allows specifying a `minDomains` property to inform the scheduler of the number of eligible domains. +`kube-scheduler` is only aware of topology domains via nodes that exist with those labels. If the above deployment is deployed to a cluster with nodes only in a single zone, all of the pods will schedule on those nodes as `kube-scheduler` isn’t aware of the other zones. For this topology spread to work as expected with the scheduler, nodes must already exist in all zones. The `minDomains` property of a topology spread constraints is used to inform the scheduler of the number of eligible domains, even if there is a Node running there to avoid this issue. @@ -125,34 +107 @@ Do not require that pods be scheduled across distinct AZs otherwise, the number -### Ensure capacity in each AZ when using EBS volumes - -If you use [Amazon EBS to provide Persistent Volumes](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html), then you need to ensure that the pods and associated EBS volume are located in the same AZ. At the time of writing, EBS volumes are only available within a single AZ. A Pod cannot access EBS-backed persistent volumes located in a different AZ. Kubernetes [scheduler knows which AZ a worker node](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone) is located in. Kubernetes will always schedule a Pod that requires an EBS volume in the same AZ as the volume. However, if there are no worker nodes available in the AZ where the volume is located, then the Pod cannot be scheduled. - -Create Auto Scaling Group for each AZ with enough capacity to ensure that the cluster always has capacity to schedule pods in the same AZ as the EBS volumes they need. In addition, you should enable the `--balance-similar-node-groups` feature in Cluster Autoscaler. - -If you are running an application that uses EBS volume but has no requirements to be highly available, then you can restrict the deployment of the application to a single AZ. In EKS, worker nodes are automatically added `failure-domain.beta.kubernetes.io/zone` label, which contains the name of the AZ. You can see the labels attached to your nodes by running `kubectl get nodes --show-labels`. More information about built-in node labels is available [here](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#built-in-node-labels). You can use node selectors to schedule a pod in a particular AZ. - -In the example below, the pod will only be scheduled in `us-west-2c` AZ: - - - apiVersion: v1 - kind: Pod - metadata: - name: single-az-pod - spec: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: failure-domain.beta.kubernetes.io/zone - operator: In - values: - - us-west-2c - containers: - - name: single-az-container - image: kubernetes/pause - -Persistent volumes (backed by EBS) are also automatically labeled with the name of AZ; you can see which AZ your persistent volume belongs to by running `kubectl get pv -L topology.ebs.csi.aws.com/zone`. When a pod is created and claims a volume, Kubernetes will schedule the Pod on a node in the same AZ as the volume. - -Consider this scenario; you have an EKS cluster with one node group. This node group has three worker nodes spread across three AZs. You have an application that uses an EBS-backed Persistent Volume. When you create this application and the corresponding volume, its Pod gets created in the first of the three AZs. Then, the worker node that runs this Pod becomes unhealthy and subsequently unavailable for use. Cluster Autoscaler will replace the unhealthy node with a new worker node; however, because the autoscaling group spans across three AZs, the new worker node may get launched in the second or the third AZ, but not in the first AZ as the situation demands. As the AZ-constrained EBS volume only exists in the first AZ, but there are no worker nodes available in that AZ, the Pod cannot be scheduled. Therefore, you should create one node group in each AZ, so there is always enough capacity available to run pods that cannot be scheduled in other AZs. - -Alternatively, [EFS](https://github.com/kubernetes-sigs/aws-efs-csi-driver) can simplify cluster autoscaling when running applications that need persistent storage. Clients can access EFS file systems concurrently from all the AZs in the region. Even if a Pod using EFS-backed Persistent Volume gets terminated and gets scheduled in different AZ, it will be able to mount the volume. +### Ensure ability to launch Nodes in each AZ when using EBS volumes @@ -160 +109 @@ Alternatively, [EFS](https://github.com/kubernetes-sigs/aws-efs-csi-driver) can -### Detect node problems with node monitoring agent +If you use Amazon EBS to provide Persistent Volumes, then you need to ensure that the pods and associated EBS volume are located in the same AZ. A Pod cannot access EBS-backed persistent volumes located in a different AZ. The Kubernetes [scheduler knows which AZ a worker node](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#topologykubernetesiozone) is located in from the labels that are on the Node and will always schedule a Pod that requires an EBS volume in the same AZ as the volume. However, if there are no worker nodes available in the AZ where the volume is located, then the Pod cannot be scheduled. @@ -162 +111 @@ Alternatively, [EFS](https://github.com/kubernetes-sigs/aws-efs-csi-driver) can -Failures in worker nodes can impact the availability of your applications. You can use the node monitoring agent to detect and show health issues. You can also enable node auto repair to automatically replace nodes when issues are detected. +If using EKS Auto Mode or Karpenter you will need to ensure that your NodeClass selects subnets in each AZ. If using Managed Node Groups, you need to ensure that you have a Node Group in each AZ. @@ -164 +113 @@ Failures in worker nodes can impact the availability of your applications. You c -The node monitoring agent is included as a capability for all Amazon EKS Auto Mode clusters. For other cluster types, you can add the monitoring agent as an Amazon EKS add-on. For more information, see [Enable node auto repair and investigate node health issues](https://docs.aws.amazon.com/eks/latest/userguide/node-health.html) in the _Amazon EKS User Guide_. +An EBS storage capability is built into EKS Auto Mode, but if using Karpenter or Managed Node Groups the [EBS CSI](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) will also need to be installed. @@ -166 +115 @@ The node monitoring agent is included as a capability for all Amazon EKS Auto Mo -### Reserve resources for system and Kubernetes daemons +### Use EKS Auto Mode to manage worker nodes @@ -168 +117 @@ The node monitoring agent is included as a capability for all Amazon EKS Auto Mo -You can improve worker nodes' stability by [reserving compute capacity for the operating system and Kubernetes daemons](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/). Pods \- especially ones without `limits` declared - can saturate system resources putting nodes in a situation where operating system processes and Kubernetes daemons (`kubelet`, container runtime, etc.) compete with pods for system resources. You can use `kubelet` flags `--system-reserved` and `--kube-reserved` to reserve resources for system process (`udev`, `sshd`, etc.) and Kubernetes daemons respectively. +EKS Auto Mode streamlines EKS management by providing production-ready clusters with minimal operational overhead. Auto Mode is responsible for scaling the number of Nodes up or down depending on the Pods that are running in the cluster. Nodes are kept up to date with software patches and fixes automatically, with the updates being performed in accordance with the configured [NodePool](https://docs.aws.amazon.com/eks/latest/userguide/create-node-pool.html#_disruption) disruption settings and Pod Disruption Budgets. @@ -170 +119 @@ You can improve worker nodes' stability by [reserving compute capacity for the o -If you use the [EKS-optimized Linux AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html), the CPU, memory, and storage are reserved for the system and Kubernetes daemons by default. When worker nodes based on this AMI launch, EC2 user-data is configured to trigger the [`bootstrap.sh` script](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh). This script calculates CPU and memory reservations based on the number of CPU cores and total memory available on the EC2 instance. The calculated values are written to the `KubeletConfiguration` file located at `/etc/kubernetes/kubelet/kubelet-config.json`. +### Run the Node Monitoring Agent @@ -172 +121 @@ If you use the [EKS-optimized Linux AMI](https://docs.aws.amazon.com/eks/latest/ -You may need to increase the system resource reservation if you run custom daemons on the node and the amount of CPU and memory reserved by default is insufficient. +The [Node Monitoring Agent](https://docs.aws.amazon.com/eks/latest/userguide/node-health.html) monitors and reacts to Node health issues by publishing Kubernetes events and updating the status condition on Nodes. The Node Monitoring Agent is included with EKS Auto Mode Nodes, and can be installed as an EKS Addon for Nodes that aren’t managed by Auto Mode. @@ -174 +123 @@ You may need to increase the system resource reservation if you run custom daemo -`eksctl` offers the easiest way to customize [resource reservation for system and Kubernetes daemons](https://eksctl.io/usage/customizing-the-kubelet/). +EKS Auto Mode, Managed Node Groups, and Karpenter all have the ability to detect fatal Node conditions reported by the Node Monitoring Agent and repair those Nodes automatically when those conditions occur. @@ -224,11 +173 @@ Consider using `LimitRange` in conjunction with `ResourceQuota` to enforce limit -## CoreDNS - -CoreDNS fulfills name resolution and service discovery functions in Kubernetes. It is installed by default on EKS clusters. For interoperability, the Kubernetes Service for CoreDNS is still named [kube-dns](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/). CoreDNS Pods run as part of a Deployment in `kube-system` namespace, and in EKS, by default, it runs two replicas with declared requests and limits. DNS queries are sent to the `kube-dns` Service that runs in the `kube-system` Namespace. - -## Recommendations - -### Monitor CoreDNS metrics - -CoreDNS has built in support for [Prometheus](https://github.com/coredns/coredns/tree/master/plugin/metrics). You should especially consider monitoring CoreDNS latency (`coredns_dns_request_duration_seconds_sum`, before [1.7.0](https://github.com/coredns/coredns/blob/master/notes/coredns-1.7.0.md) version the metric was called `core_dns_response_rcode_count_total`), errors (`coredns_dns_responses_total`, NXDOMAIN, SERVFAIL, FormErr) and CoreDNS Pod’s memory consumption. - -For troubleshooting purposes, you can use kubectl to view CoreDNS logs: +### Use NodeLocal DNSCache @@ -235,0 +175 @@ For troubleshooting purposes, you can use kubectl to view CoreDNS logs: +You can improve the Cluster DNS performance by running [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/). This feature runs a DNS caching agent on cluster nodes as a DaemonSet. All the pods use the DNS caching agent running on the node for name resolution instead of using `kube-dns` Service. This feature is automatically included in EKS Auto Mode. @@ -237 +177 @@ For troubleshooting purposes, you can use kubectl to view CoreDNS logs: - for p in $(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[*].metadata.name}'); do kubectl logs $p -n kube-system; done +### Configure auto-scaling CoreDNS @@ -239 +179 @@ For troubleshooting purposes, you can use kubectl to view CoreDNS logs: -### Use NodeLocal DNSCache +Another method of improving Cluster DNS performance is by enabling the built-in [auto-scaling of CoreDNS Pods](https://docs.aws.amazon.com/eks/latest/userguide/coredns-autoscaling.html). @@ -241,61 +181 @@ For troubleshooting purposes, you can use kubectl to view CoreDNS logs: -You can improve the Cluster DNS performance by running [NodeLocal DNSCache](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/). This feature runs a DNS caching agent on cluster nodes as a DaemonSet. All the pods use the DNS caching agent running on the node for name resolution instead of using `kube-dns` Service. - -### Configure cluster-proportional-scaler for CoreDNS - -Another method of improving Cluster DNS performance is by [automatically horizontally scaling the CoreDNS Deployment](https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/#enablng-dns-horizontal-autoscaling) based on the number of nodes and CPU cores in the cluster. [Horizontal cluster-proportional-autoscaler](https://github.com/kubernetes-sigs/cluster-proportional-autoscaler/blob/master/README.md) is a container that resizes the number of replicas of a Deployment based on the size of the schedulable data-plane. - -Nodes and the aggregate of CPU cores in the nodes are the two metrics with which you can scale CoreDNS. You can use both metrics simultaneously. If you use larger nodes, CoreDNS scaling is based on the number of CPU cores. Whereas, if you use smaller nodes, the number of CoreDNS replicas depends on the CPU cores in your data-plane. Proportional autoscaler configuration looks like this: - - - linear: '{"coresPerReplica":256,"min":1,"nodesPerReplica":16}' - -### Choosing an AMI with Node Group - -EKS provides optimized EC2 AMIs that are used by customers to create both self-managed and managed nodegroups. These AMIs are published in every region for every supported Kubernetes version. EKS marks these AMIs as deprecated when any CVEs or bugs are discovered. Hence, the recommendation is not to consume deprecated AMIs while choosing an AMI for the node group. - -Deprecated AMIs can be filtered using Ec2 describe-images api using below command: - - - aws ec2 describe-images --image-id ami-0d551c4f633e7679c --no-include-deprecated - -You can also recognize a deprecated AMI by verifying if the describe-image output contains a DeprecationTime as a field. For ex: - - - aws ec2 describe-images --image-id ami-xxx --no-include-deprecated - { - "Images": [ - { - "Architecture": "x86_64", - "CreationDate": "2022-07-13T15:54:06.000Z", - "ImageId": "ami-xxx", - "ImageLocation": "123456789012/eks_xxx", - "ImageType": "machine", - "Public": false, - "OwnerId": "123456789012", - "PlatformDetails": "Linux/UNIX", - "UsageOperation": "RunInstances", - "State": "available", - "BlockDeviceMappings": [ - { - "DeviceName": "/dev/xvda", - "Ebs": { - "DeleteOnTermination": true, - "SnapshotId": "snap-0993a2fc4bbf4f7f4", - "VolumeSize": 20, - "VolumeType": "gp2", - "Encrypted": false - } - } - ], - "Description": "EKS Kubernetes Worker AMI with AmazonLinux2 image, (k8s: 1.19.15, docker: 20.10.13-2.amzn2, containerd: 1.4.13-3.amzn2)", - "EnaSupport": true, - "Hypervisor": "xen", - "Name": "aws_eks_optimized_xxx", - "RootDeviceName": "/dev/xvda", - "RootDeviceType": "ebs", - "SriovNetSupport": "simple", - "VirtualizationType": "hvm", - "DeprecationTime": "2023-02-09T19:41:00.000Z" - } - ] - } +This feature continuously monitors the cluster state, including the number of nodes and CPU cores. Based on that information, the controller will dynamically adapt the number of replicas of the CoreDNS deployment in an EKS cluster.