AWS eks documentation change
Summary
Added section 'View Karpenter events in control plane logs' with CloudWatch query to monitor Karpenter-related events for troubleshooting node provisioning, scaling, and termination issues
Security assessment
The change adds operational troubleshooting guidance for Karpenter events but does not address security vulnerabilities or document security-specific features. The events monitored relate to cluster operations rather than security incidents.
Diff
diff --git a/eks/latest/userguide/auto-troubleshoot.md b/eks/latest/userguide/auto-troubleshoot.md index 8278f3cdd..804ff1497 100644 --- a//eks/latest/userguide/auto-troubleshoot.md +++ b//eks/latest/userguide/auto-troubleshoot.md @@ -5 +5 @@ -Node monitoring agentGet console output from an EC2 managed instance by using the AWS EC2 CLIGet node logs by using debug containers and the kubectl CLIView resources associated with EKS Auto Mode in the AWS ConsoleView IAM Errors in your AWS accountTroubleshoot Pod failing to schedule onto Auto Mode nodeTroubleshoot node not joining the clusterSharing Volumes Across PodsTroubleshoot included controllers in Auto Mode +Node monitoring agentGet console output from an EC2 managed instance by using the AWS EC2 CLIGet node logs by using debug containers and the kubectl CLIView resources associated with EKS Auto Mode in the AWS ConsoleView IAM Errors in your AWS accountTroubleshoot Pod failing to schedule onto Auto Mode nodeTroubleshoot node not joining the clusterSharing Volumes Across PodsView Karpenter events in control plane logsTroubleshoot included controllers in Auto Mode @@ -284,0 +285,65 @@ To enable this sharing between Pods, you can use the Pod’s `seLinuxOptions` to +## View Karpenter events in control plane logs + +For EKS clusters with control plane logs enabled, you can gain insights into Karpenter’s actions and decision-making process by querying the logs. This can be particularly useful for troubleshooting EKS Auto Mode issues related to node provisioning, scaling, and termination. To view Karpenter-related events, use the following CloudWatch Logs Insights query: + + + fields @timestamp, @message + | filter @logStream like /kube-apiserver-audit/ + | filter @message like 'DisruptionBlocked' + or @message like 'DisruptionLaunching' + or @message like 'DisruptionTerminating' + or @message like 'DisruptionWaitingReadiness' + or @message like 'Unconsolidatable' + or @message like 'FailedScheduling' + or @message like 'NoCompatibleInstanceTypes' + or @message like 'NodeRepairBlocked' + or @message like 'Disrupted' + or @message like 'Evicted' + or @message like 'FailedDraining' + or @message like 'TerminationGracePeriodExpiring' + or @message like 'TerminationFailed' + or @message like 'FailedConsistencyCheck' + or @message like 'InsufficientCapacityError' + or @message like 'UnregisteredTaintMissing' + or @message like 'NodeClassNotReady' + sort @timestamp desc + +This query filters for specific [Karpenter-related events](https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/events/reason.go) in the kube-apiserver audit logs. The events include various disruption states, scheduling failures, capacity issues, and node-related problems. By analyzing these logs, you can gain a better understanding of: + + * Why Karpenter is taking certain actions. + + * Any issues preventing proper node provisioning, scaling, or termination. + + * Potential capacity or compatibility problems with instance types. + + * Node lifecycle events such as disruptions, evictions, or terminations. + + + + +To use this query: + + 1. Navigate to the CloudWatch console + + 2. Select "Logs Insights" from the left navigation pane + + 3. Choose the log group for your EKS cluster’s control plane logs + + 4. Paste the query into the query editor + + 5. Adjust the time range as needed + + 6. Run the query + + + + +The results will show you a timeline of Karpenter-related events, helping you troubleshoot issues, and understand the behavior of EKS Auto Mode in your cluster. To review Karpenter actions on a specific node, you can add the below line filter specifying the instance ID to the aforementioned query: + + + |filter @message like /[.replaceable]`i-12345678910123456`/ + +###### Note + +To use this query, control plane logging must be enabled on your EKS cluster. If you haven’t done this yet, please refer to [Send control plane logs to CloudWatch Logs](./control-plane-logs.html). +