AWS eks: Add credential exposure warning for NodeDiagnostic pre-signed URLs
Summary
Expands EKS node log retrieval documentation, adds an Important warning that NodeDiagnostic spec stores pre-signed S3 credentials readable via kubectl, and documents the node destination alternative.
Security assessment
The added Important note explicitly warns that NodeDiagnostic spec contains pre-signed S3 credentials readable by anyone with kubectl get/describe, and advises short expiration and restricted access. This documents a credential-exposure weakness and hardening guidance.
Evidence
The `spec` of the `NodeDiagnostic` resource stores the destination URL in full, including the pre-signed credentials. Anyone who can read the resource with `kubectl get` or `kubectl describe` can read those credentials and write to that S3 object until the URL expires. Keep the expiration short and restrict access to `NodeDiagnostic` resources accordingly.
Diff
diff --git a/eks/latest/userguide/auto-get-logs.md b/eks/latest/userguide/auto-get-logs.md index cfea968..5bb229d 100644 --- a//eks/latest/userguide/auto-get-logs.md +++ b//eks/latest/userguide/auto-get-logs.md @@ -7 +7 @@ -PrerequisitesStep 1: Create S3 bucket destination (optional)Step 2: Create pre-signed S3 URL for HTTP PutStep 3: Create NodeDiagnostic resourceStep 4: Download logs from S3Step 5: Clean up NodeDiagnostic resourceNodeDiagnostic node DestinationExample Usage +PrerequisitesStep 1: Create S3 bucket destination (optional)Step 2: Create pre-signed S3 URL for HTTP PutStep 3: Create NodeDiagnostic resourceStep 4: Download logs from S3Step 5: Clean up NodeDiagnostic resourceNodeDiagnostic node destinationExample Usage @@ -13 +13 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -# Retrieve node logs for a managed node using kubectl and S3 +# Retrieve node logs using kubectl and Amazon S3 @@ -15 +15,5 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -Learn how to retrieve node logs for an Amazon EKS managed node that has the node monitoring agent. +Learn how to retrieve node logs from an Amazon EKS node that runs the EKS node monitoring agent. The agent collects log and configuration bundles from the node and uploads them to your S3 bucket. + +Log collection works on any EKS node that runs the EKS node monitoring agent, regardless of how the node is provisioned. This includes EKS Auto Mode nodes, EKS managed node groups, self-managed nodes, and Karpenter-provisioned nodes. AWS Fargate doesn’t support the EKS node monitoring agent, so you can’t collect node logs this way for Fargate pods. + +The agent provides the `NodeDiagnostic` custom resource that this procedure uses. EKS Auto Mode includes the agent, so the custom resource is available without any installation. For other compute types, install the EKS node monitoring agent as an EKS add-on or with Kubernetes tooling such as Helm first. For more information, see [Configure the Amazon EKS node monitoring agent](./node-monitoring-agent-configure.html). @@ -21 +25 @@ Make sure you have the following: - * An existing Amazon EKS cluster with the node monitoring agent. For more information, see [Detect node health issues and enable automatic node repair](./node-health.html). + * An existing Amazon EKS cluster with the EKS node monitoring agent running. For more information, see [Detect node health issues and enable automatic node repair](./node-health.html). @@ -75 +79,4 @@ For more information, see [Generate a presigned URL to upload a file](https://bo -Identify the name of the node you want to collect logs from. +Identify the name of the node you want to collect logs from, as reported by `kubectl get nodes`. Depending on the compute type, this is either an EC2 instance ID or a private DNS name. Use the name exactly as `kubectl` reports it. + + + kubectl get nodes @@ -94 +101 @@ Apply the manifest to the cluster. -You can check on the Status of the collection by describing the `NodeDiagnostic` resource: +You can check on the status of the collection by describing the `NodeDiagnostic` resource. Collection usually finishes within a few seconds. @@ -96 +102,0 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` - * A status of `Success` or `SuccessWithErrors` indicates that the task completed and the logs uploaded to the provided destination (`SuccessWithErrors` indicates that some logs might be missing) @@ -98 +104 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` - * If the status is Failure, confirm the upload URL is well-formed and not expired. + kubectl describe nodediagnostics.eks.amazonaws.com/<node-name> @@ -100,0 +107,11 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` +The result appears under `Status` as a capture status of type `Log`. To read just the status, use the following command. + + + kubectl get nodediagnostics.eks.amazonaws.com/<node-name> \ + -o jsonpath='{.status.captureStatuses}' + +Interpret the `reason` field as follows: + + * A `reason` of `Success` or `SuccessWithErrors` indicates that the task completed and the logs uploaded to the provided destination (`SuccessWithErrors` indicates that some logs might be missing). + + * A `reason` of `Failure` indicates that collection or upload failed. The accompanying `message` field includes the error returned by Amazon S3. Confirm that the upload URL is well-formed and hasn’t expired, then generate a new pre-signed URL and try again. @@ -104 +120,0 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` - kubectl describe nodediagnostics.eks.amazonaws.com/<node-name> @@ -105,0 +122,3 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` +###### Important + +The `spec` of the `NodeDiagnostic` resource stores the destination URL in full, including the pre-signed credentials. Anyone who can read the resource with `kubectl get` or `kubectl describe` can read those credentials and write to that S3 object until the URL expires. Keep the expiration short and restrict access to `NodeDiagnostic` resources accordingly. @@ -109 +128 @@ You can check on the Status of the collection by describing the `NodeDiagnostic` -Wait approximately one minute before attempting to download the logs. Then, use the S3 CLI to download the logs. +Wait until the `NodeDiagnostic` resource reports a `reason` of `Success` or `SuccessWithErrors`. Then, use the S3 CLI to download the logs. @@ -127 +146,14 @@ Wait approximately one minute before attempting to download the logs. Then, use -## NodeDiagnostic `node` Destination +## NodeDiagnostic `node` destination + +Starting with version `v1.6.1-eksbuild.1` of the EKS node monitoring agent, you can set the log collection destination to `node` instead of a pre-signed URL. This collects the logs and persists them temporarily on the node for later collection, so you don’t need an S3 bucket or pre-signed URL. + + + apiVersion: eks.amazonaws.com/v1alpha1 + kind: NodeDiagnostic + metadata: + name: <node-name> + spec: + logCapture: + destination: node + +The EKS node monitoring agent writes the bundle to `/var/log/support/[.replaceable]`<node-name>`-logs.tar.gz` on the node, and the `message` field in the capture status reports the path it used. This destination works on all compute types that support the agent, including EKS Auto Mode and EKS managed node groups. @@ -129 +161 @@ Wait approximately one minute before attempting to download the logs. Then, use -Starting with version `v1.6.1-eksbuild.1` of the Node Monitoring Agent, there is an option to set the log collection destination to `node`. Using this destination will lead to the collection and temporary persistence of logs on the node for later collection. In addition to this functionality, within the Node Monitoring Agent’s GitHub repository is a `kubectl` plugin you can install for easy interaction and log collection. For more information, see the [documentation for the `kubectl ekslogs` plugin](https://github.com/aws/eks-node-monitoring-agent/blob/main/tools/kubectl-ekslogs/README.md). +The EKS node monitoring agent’s GitHub repository also includes a `kubectl` plugin that creates the `NodeDiagnostic` resource and retrieves the resulting bundles for you. For more information, see the [documentation for the `kubectl ekslogs` plugin](https://github.com/aws/eks-node-monitoring-agent/blob/main/tools/kubectl-ekslogs/README.md).