AWS eks documentation change
Summary
Restructured node health documentation with updated terminology, added node condition table, removed detailed health issue tables, and consolidated automatic repair information.
Security assessment
The changes focus on documentation reorganization and clarification of node health monitoring features. No security vulnerabilities, weaknesses, or incidents are mentioned. The updates describe operational health monitoring without introducing security-specific content or addressing security flaws.
Diff
diff --git a/eks/latest/userguide/node-health.md b/eks/latest/userguide/node-health.md index ceb57597c..627e5bb07 100644 --- a//eks/latest/userguide/node-health.md +++ b//eks/latest/userguide/node-health.md @@ -5 +5 @@ -Node monitoring agentNode auto repairNode health issues +Node monitoring agentAutomatic node repair @@ -11 +11 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -# Enable node auto repair and investigate node health issues +# Detect node health issues and enable automatic node repair @@ -13 +13 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -Node health refers to the operational status and capability of a node to effectively run workloads. A healthy node maintains expected connectivity, has sufficient resources, and can successfully run Pods without disruption. For information on getting details about your nodes, see [View the health status of your nodes](./learn-status-conditions.html) and [Retrieve node logs for a managed node using kubectl and S3](./auto-get-logs.html). +Node health refers to the operational status and capability of a Kubernetes node to effectively run workloads. A healthy node maintains expected network connectivity, has sufficient compute and storage resources, and can successfully run workloads without disruption. @@ -15 +15 @@ Node health refers to the operational status and capability of a node to effecti -To help with maintaining healthy nodes, Amazon EKS offers the _node monitoring agent_ and _node auto repair_. +To help with maintaining healthy nodes in EKS clusters, EKS offers the _node monitoring agent_ and _automatic node repair_. These features are automatically enabled with EKS Auto Mode compute. You can also use automatic node repair with EKS managed node groups and Karpenter, and can use the EKS node monitoring agent with any EKS compute types except for AWS Fargate. The EKS node monitoring agent and automatic node repair are most effective when used together, but they can also be used individually in EKS clusters. @@ -23 +23 @@ The _node monitoring agent_ and _node auto repair_ are only available on Linux. -The node monitoring agent automatically reads node logs to detect certain health issues. It parses through node logs to detect failures and surfaces various status information about worker nodes. A dedicated `NodeCondition` is applied on the worker nodes for each category of issues detected, such as storage and networking issues. Descriptions of detected health issues are made available in the observability dashboard. For more information, see [Node health issues](./observability-dashboard.html#observability-node-health-issues). +The EKS node monitoring agent reads node logs to detect health issues. It parses logs to detect failures and surfaces status information about the health status of the nodes. For each category of issues detected, the agent applies a dedicated `NodeCondition` to the worker nodes. For detailed information on the node health issues detected by the EKS node monitoring agent, see [Detect node health issues with the EKS node monitoring agent](./node-health-nma.html). @@ -25 +25 @@ The node monitoring agent automatically reads node logs to detect certain health -The node monitoring agent is included as a capability for all Amazon EKS Auto Mode clusters. For other cluster types, you can add the monitoring agent as an Amazon EKS add-on. For more information, see [Create an Amazon EKS add-on](./creating-an-add-on.html). +EKS Auto Mode compute includes the node monitoring agent. For other EKS compute types, you can add the node monitoring agent as an EKS add-on or you can manage it with Kubernetes tooling such as Helm. For more information, see [Configure the node monitoring agent](./node-health-nma.html#node-monitoring-agent-configure). @@ -27 +27 @@ The node monitoring agent is included as a capability for all Amazon EKS Auto Mo -## Node auto repair +With the EKS node monitoring agent, the following categories of node health issues are surfaced as node conditions. Note, `Ready`, `DiskPressure`, and `MemoryPressure` are standard Kubernetes node conditions that are surfaced even without the EKS node monitoring agent. @@ -29 +29,10 @@ The node monitoring agent is included as a capability for all Amazon EKS Auto Mo -Node auto repair is an additional feature that continuously monitors the health of nodes, automatically reacting to detected problems and replacing nodes when possible. This helps overall availability of the cluster with minimal manual intervention. If a health check fails, the node is automatically cordoned so that no new Pods are scheduled on the node. +Node Condition | Description +---|--- +AcceleratedHardwareReady | AcceleratedHardwareReady indicates whether accelerated hardware (GPU, Neuron) on the node is functioning correctly. +ContainerRuntimeReady | ContainerRuntimeReady indicates whether the container runtime (containerd, etc.) is functioning correctly and able to run containers. +DiskPressure | DiskPressure is a standard Kubernetes condition indicating the node is experiencing disk pressure (low disk space or high I/O). +KernelReady | KernelReady indicates whether the kernel is functioning correctly without critical errors, panics, or resource exhaustion. +MemoryPressure | MemoryPressure is a standard Kubernetes condition indicating the node is experiencing memory pressure (low available memory). +NetworkingReady | NetworkingReady indicates whether the node’s networking stack is functioning correctly (interfaces, routing, connectivity). +StorageReady | StorageReady indicates whether the node’s storage subsystem is functioning correctly (disks, filesystems, I/O). +Ready | Ready is the standard Kubernetes condition indicating the node is healthy and ready to accept pods. @@ -31 +40 @@ Node auto repair is an additional feature that continuously monitors the health -By itself, node auto repair can react to the `Ready` condition of the `kubelet` and any node objects that are manually deleted. When paired with the node monitoring agent, node auto repair can react to more conditions that wouldn’t be detected otherwise. These additional conditions include `KernelReady`, `NetworkingReady`, and `StorageReady`. +## Automatic node repair @@ -33 +42 @@ By itself, node auto repair can react to the `Ready` condition of the `kubelet` -This automated node recovery automatically addresses intermittent node issues such as failures to join the cluster, unresponsive kubelets, and increased accelerator (device) errors. The improved reliability helps reduce application downtime and improve cluster operations. By default, node auto repair does not automatically repair nodes for certain conditions such as `DiskPressure`, `MemoryPressure`, `PIDPressure`, and DCGM (NVIDIA Data Center GPU Manager) diagnostic or monitoring tool errors. These conditions often indicate issues with application behavior, workload configuration, or resource limits rather than node-level failures, making it difficult to determine an appropriate default repair action. However, you can customize this behavior using `nodeRepairConfigOverrides` to enable automatic repair actions for these conditions based on your use case. Amazon EKS waits 10 minutes before acting on the `AcceleratedHardwareReady` `NodeConditions`, and 30 minutes for all other conditions. +EKS automatic node repair continuously monitors node health, reacts to detected problems, and replaces or reboots nodes when possible. This improves cluster reliability with minimal manual intervention and helps reduce application downtime. @@ -35 +44 @@ This automated node recovery automatically addresses intermittent node issues su -Managed node groups will also automatically disable node repairs for safety reasons under two scenarios. Any repair operations that are previously in progress will continue for both situations. +By itself, EKS automatic node repair reacts to the `Ready` conditions of the kubelet, any manually deleted node objects, and EKS managed node group instances that fail to join the cluster. When EKS automatic node repair is enabled with the node monitoring agent installed, EKS automatic node repair reacts to additional node conditions: `AcceleratedHardwareReady`, `ContainerRuntimeReady`, `KernelReady`, `NetworkingReady`, and `StorageReady`. @@ -37 +46 @@ Managed node groups will also automatically disable node repairs for safety reas - * If a zonal shift for your cluster has been triggered through the Application Recovery Controller (ARC), all subsequent repair operations are halted. +EKS automatic node repair does not react to standard Kubernetes `DiskPressure`, `MemoryPressure`, or `PIDPressure` node conditions. These conditions often indicate issues with application behavior, workload configuration, or resource limits rather than node-level failures, making it difficult to determine an appropriate default repair action. In these scenarios, workloads are subject to the Kubernetes [node pressure eviction behavior](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction). @@ -39 +48 @@ Managed node groups will also automatically disable node repairs for safety reas - * If your node group has more than five nodes and more than 20% of the nodes in your node group are in an unhealthy state, repair operations are halted. +For more information on EKS automatic node repair, see [Automatically repair nodes in EKS clusters](./node-repair.html). @@ -40,0 +50 @@ Managed node groups will also automatically disable node repairs for safety reas +###### Topics @@ -44,169 +53,0 @@ Managed node groups will also automatically disable node repairs for safety reas -You can enable node auto repair when creating or editing a managed node group. - - * When using the Amazon EKS console, activate the **Enable node auto repair** checkbox for the managed node group. For more information, see [Create a managed node group for your cluster](./create-managed-node-group.html). - - * When using the AWS CLI, add the `--node-repair-config enabled=true` to the [`eks create nodegroup`](https://docs.aws.amazon.com/cli/latest/reference/eks/create-nodegroup.html) or [`eks update-nodegroup-config`](https://docs.aws.amazon.com/cli/latest/reference/eks/update-nodegroup-config.html) command. - - * For an example `eksctl` `ClusterConfig` that uses a managed node group with node auto repair, see [44-node-repair.yaml](https://github.com/eksctl-io/eksctl/blob/main/examples/44-node-repair.yaml) on GitHub. - - - - -Amazon EKS provides more granular control over the node auto repair behavior through the following: - - * `maxUnhealthyNodeThresholdCount` and `maxUnhealthyNodeThresholdPercentage` - - * These fields allow you to specify a count or percentage threshold of unhealthy nodes, above which node auto repair actions will stop. This provides more control over the "blast radius" of node auto repairs. - - * You can set either the absolute count or percentage, but not both. - - * `maxParallelNodesRepairedCount` and `maxParallelNodesRepairedPercentage` - - * These fields allow you to specify the maximum number of nodes that can be repaired concurrently or in parallel, expressed as either a count or percentage of all unhealthy nodes. This gives you finer-grained control over the pace of node replacements. - - * As with the unhealthy node threshold, you can set either the absolute count or percentage, but not both. - - * `nodeRepairConfigOverrides` - - * This is a complex structure that allows you to set granular overrides for specific repair actions. These overrides control the repair action and the repair delay time before a node is considered eligible for repair. - - * The specific fields in this structure are: - - * `nodeMonitoringCondition`: The unhealthy condition reported by the node monitoring agent. - - * `nodeUnhealthyReason`: The reason why the node monitoring agent identified the node as unhealthy. - - * `minRepairWaitTimeMins`: The minimum time (in minutes) that the repair condition and unhealthy reason must persist before the node is eligible for repair. - - * `repairAction`: The action the repair system should take when the above conditions are met. - - * If you use this field, you must specify all the fields in the structure. You can also provide a list of these overrides. - - * The `nodeMonitoringCondition` and `nodeUnhealthyReason` are manual text inputs that you set to indicate you want to deviate from the system’s default behavior. - - * The `minRepairWaitTimeMins` and `repairAction` fields allow you to specify deviations from the system’s default behavior. - - * The following example shows how to override the wait time to 20 minutes before Amazon EKS reboots a node experiencing `NvidiaXID13Error` conditions. By default, Amazon EKS waits 10 minutes before taking repair action on `AcceleratedHardwareReady` conditions. - - aws eks update-nodegroup-config \ - --cluster-name my-cluster \ - --nodegroup-name my-nodegroup \ - --node-repair-config 'enabled=true,nodeRepairConfigOverrides=[{nodeMonitoringCondition=AcceleratedHardwareReady,nodeUnhealthyReason=NvidiaXID13Error,minRepairWaitTimeMins=20}]' - - - - -## Node health issues - -The following tables describe node health issues that can be detected by the node monitoring agent. There are two types of issues: - - * Condition – A terminal issue that warrants a remediation action like an instance replacement or reboot. When auto repair is enabled, Amazon EKS will do a repair action, either as a node replacement or reboot. For more information, see [Node conditions](./learn-status-conditions.html#status-node-conditions). - - * Event – A temporary issue or sub-optimal node configuration. No auto repair action will take place. For more information, see [Node events](./learn-status-conditions.html#status-node-events). - - - - -### AcceleratedHardware node health issues - -The monitoring condition is `AcceleratedHardwareReady` for issues in the following table that have a severity of “Condition”. - -If auto repair is enabled, the repair actions that are listed start 10 minutes after the issue is detected. For more information on XID errors, see [Xid Errors](https://docs.nvidia.com/deploy/xid-errors/index.html#topic_5_1) in the _NVIDIA GPU Deployment and Management Documentation_. For more information on the individual XID messages, see [Understanding Xid Messages](https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html#understanding-xid-messages) in the _NVIDIA GPU Deployment and Management Documentation_. - -Name | Severity | Description | Repair Action ----|---|---|--- -DCGMDiagnosticFailure | Condition | A test case from the DCGM active diagnostics test suite failed. | None -DCGMError | Condition | Connection to the DCGM host process was lost or could not be established. | None -DCGMFieldError[Code] | Event | DCGM detected GPU degradation through a field identifier. | None -DCGMHealthCode[Code] | Event | A DCGM health check failed in a non-fatal manner. | None -DCGMHealthCode[Code] | Condition | A DCGM health check failed in a fatal manner. | None -NeuronDMAError | Condition | A DMA engine encountered an unrecoverable error. | Replace -NeuronHBMUncorrectableError | Condition | An HBM encountered an uncorrectable error and produced incorrect results. | Replace -NeuronNCUncorrectableError | Condition | A Neuron Core uncorrectable memory error was detected. | Replace -NeuronSRAMUncorrectableError | Condition | An on-chip SRAM encountered a parity error and produced incorrect results. | Replace -NvidiaDeviceCountMismatch | Event | The number of GPUs visible through NVML is inconsistent with the NVIDIA device count on the filesystem. | None -NvidiaDoubleBitError | Condition | A double bit error was produced by the GPU driver. | Replace -NvidiaNCCLError | Event | A segfault occurred in the NVIDIA Collective Communications library (`libnccl`). | None -NvidiaNVLinkError | Condition | NVLink errors were reported by the GPU driver. | Replace -NvidiaPCIeError | Event | PCIe replays were triggered to recover from transmission errors. | None -NvidiaPageRetirement | Event | The GPU driver has marked a memory page for retirement. This may occur if there is a single double bit error or two single bit errors are encountered at the same address. | None -NvidiaPowerError | Event | Power utilization of GPUs breached the allowed thresholds. | None -NvidiaThermalError | Event | Thermal status of GPUs breached the allowed thresholds. | None -NvidiaXID[Code]Error | Condition | A critical GPU error occurred. | Replace or Reboot -NvidiaXID[Code]Warning | Event | A non-critical GPU error occurred. | None - -### ContainerRuntime node health issues - -The monitoring condition is `ContainerRuntimeReady` for issues in the following table that have a severity of “Condition”. - -Name | Severity | Description | Repair Action ----|---|---|--- -ContainerRuntimeFailed | Event | The container runtime has failed to create a container, likely related to any reported issues if occurring repeatedly. | None -DeprecatedContainerdConfiguration | Event | A container image using deprecated image manifest version 2, schema 1 was recently pulled onto the node through `containerd`. | None -KubeletFailed | Event | The kubelet entered a failed state. | None -LivenessProbeFailures | Event | A liveness probe failure was detected, potentially indicating application code issues or insufficient timeout values if occurring repeatedly. | None -PodStuckTerminating | Condition | A Pod is or was stuck terminating for an excessive amount of time, which can be caused by CRI errors preventing pod state progression. | Replace -ReadinessProbeFailures | Event | A readiness probe failure was detected, potentially indicating application code issues or insufficient timeout values if occurring repeatedly. | None -[Name]RepeatedRestart | Event | A systemd unit is restarting frequently. | None -ServiceFailedToStart | Event | A systemd unit failed to start. | None - -### Kernel node health issues - -The monitoring condition is `KernelReady` for issues in the following table that have a severity of “Condition”. - -Name | Severity | Description | Repair Action ----|---|---|--- -AppBlocked | Event | The task has been blocked for a long period of time from scheduling, usually caused by being blocked on input or output. | None -AppCrash | Event | An application on the node has crashed. | None -ApproachingKernelPidMax | Event | The number of processes is approaching the maximum number of PIDs that are available per the current `kernel.pid_max` setting, after which no more processes can be launched. | None -ApproachingMaxOpenFiles | Event | The number of open files is approaching the maximum number of possible open files given the current kernel settings, after which opening new files will fail. | None -ConntrackExceededKernel | Event | Connection tracking exceeded the maximum for the kernel and new connections could not be established, which can result in packet loss. | None -ExcessiveZombieProcesses | Event | Processes which can’t be fully reclaimed are accumulating in large numbers, which indicates application issues and may lead to reaching system process limits. | None -ForkFailedOutOfPIDs | Condition | A fork or exec call has failed due to the system being out of process IDs or memory, which may be caused by zombie processes or physical memory exhaustion. | Replace -KernelBug | Event | A kernel bug was detected and reported by the Linux kernel itself, though this may sometimes be caused by nodes with high CPU or memory usage leading to delayed event processing. | None -LargeEnvironment | Event | The number of environment variables for this process is larger than expected, potentially caused by many services with `enableServiceLinks` set to true, which may cause performance issues. | None -RapidCron | Event | A cron job is running faster than every five minutes on this node, which may impact performance if the job consumes significant resources. | None -SoftLockup | Event | The CPU stalled for a given amount of time. | None - -### Networking node health issues - -The monitoring condition is `NetworkingReady` for issues in the following table that have a severity of “Condition”. - -Name | Severity | Description | Repair Action ----|---|---|--- -BandwidthInExceeded | Event | Packets have been queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance. | None -BandwidthOutExceeded | Event | Packets have been queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance. | None -ConntrackExceeded | Event | Connection tracking exceeded the maximum for the instance and new connections could not be established, which can result in packet loss. | None -IPAMDInconsistentState | Event | The state of the IPAMD checkpoint on disk does not reflect the IPs in the container runtime. | None -IPAMDNoIPs | Event | IPAMD is out of IP addresses. | None -IPAMDNotReady | Condition | IPAMD fails to connect to the API server. | Replace -IPAMDNotRunning | Condition | The Amazon VPC CNI process was not found to be running. | Replace -IPAMDRepeatedlyRestart | Event | Multiple restarts in the IPAMD service have occurred. | None -InterfaceNotRunning | Condition | This interface appears to not be running or there are network issues. | Replace -InterfaceNotUp | Condition | This interface appears to not be up or there are network issues. | Replace -KubeProxyNotReady | Event | Kube-proxy failed to watch or list resources. | None -LinkLocalExceeded | Event | Packets were dropped because the PPS of traffic to local proxy services exceeded the network interface maximum. | None