AWS sagemaker high security documentation change

Service: sagemaker · 2025-08-25 · Security-related high

File: sagemaker/latest/dg/hyperpod-observability-addon-troubleshoting.md

Summary

Added detailed IAM role trust policy requirements, expanded troubleshooting steps for pod identity issues, and added new sections for reviewing observability pods and resolving pending pods

Security assessment

Added explicit IAM role configuration requirements with source ARN/account validation to prevent unauthorized access. Incorrect IAM configurations could lead to privilege escalation or unauthorized resource access.

Diff

diff --git a/sagemaker/latest/dg/hyperpod-observability-addon-troubleshoting.md b/sagemaker/latest/dg/hyperpod-observability-addon-troubleshoting.md
index feb9674a9..e303b13ba 100644
--- a//sagemaker/latest/dg/hyperpod-observability-addon-troubleshoting.md
+++ b//sagemaker/latest/dg/hyperpod-observability-addon-troubleshoting.md
@@ -5 +5 @@
-Troubleshooting missing metrics in Amazon Managed GrafanaTroubleshooting add-on installation failures
+Missing metricsAdd-on installation failuresReview all observability podsPods stuck in pending
@@ -48 +48 @@ If metrics don't appear in your Amazon Managed Grafana dashboards, perform the f
-  5. If the status isn't **ACTIVE** , copy the error message and contact AWS Support.
+  5. If the status isn't **ACTIVE** , see Troubleshooting add-on installation failures.
@@ -103,0 +104,25 @@ If metrics don't appear in your Amazon Managed Grafana dashboards, perform the f
+  6. Ensure that the IAM role that is attached to this association has the following trust policy. Verify that the source ARN and source account are correct.
+    
+        {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
+                "Effect": "Allow",
+                "Principal": {
+                    "Service": "pods.eks.amazonaws.com"
+                },
+                "Action": [
+                    "sts:AssumeRole",
+                    "sts:TagSession"
+                ],
+                "Condition": {
+                    "StringEquals": {
+                        "aws:SourceArn": "arn:aws:eks:us-east-1:111122223333:cluster/cluster-name",
+                        "aws:SourceAccount": "111122223333"
+                    }
+                }
+            }
+        ]
+    }
+
@@ -140 +165,5 @@ If the observability add-on fails to install, use the following steps to diagnos
-  6. Contact AWS Support with the issue details.
+  6. If the health issue is related to credentials or pod identity, see Verify Pod Identity association. Also ensure that the pod identity agent add-on is running in the cluster.
+
+  7. Check for errors in the manager logs. For instructions, see Review manager logs.
+
+  8. Contact AWS Support with the issue details.
@@ -149 +178 @@ If the observability add-on fails to install, use the following steps to diagnos
-        kubectl get pods -n hyperpod-observability | grep manager
+        kubectl logs -n hyperpod-observability -l control-plane=hyperpod-observability-controller-manager
@@ -154,0 +184,46 @@ If the observability add-on fails to install, use the following steps to diagnos
+  3. For urgent issues, contact Support.
+
+
+
+
+## Review all observability pods
+
+All the pods that the SageMaker HyperPod observability add-on creates are in the `hyperpod-observability` namespace. To get the status of these pods, run the following command.
+    
+    
+    kubectl get pods -n hyperpod-observability
+
+Look for the pods whose status is either `pending` or `crashloopbackoff`. Run the following command to get the logs of these pending or failing pods.
+    
+    
+    kubectl logs -n hyperpod-observability pod-name
+
+If you don't find errors in the logs, run the following command to describe the pods and look for errors.
+    
+    
+    kubectl describe -n hyperpod-observability pod pod-name
+
+To get more context, run the two following commands to describe the deployments and daemonsets for these pods.
+    
+    
+    kubectl describe -n hyperpod-observability deployment deployment-name
+    
+    
+    kubectl describe -n hyperpod-observability daemonset daemonset-name
+
+## Troubleshooting pods that are stuck in the pending status
+
+If you see that there are pods that are stuck in the `pending` status, make sure that the node is large enough to fit in all the pods. To verify that it is, perform the following steps.
+
+  1. Open the Amazon EKS console at [https://console.aws.amazon.com/eks/home#/clusters](https://console.aws.amazon.com/eks/home#/clusters).
+
+  2. Choose your cluster.
+
+  3. Choose the cluster's **Compute** tab.
+
+  4. Choose the node with the smallest instance type.
+
+  5. In the capacity allocation section, look for available pods.
+
+  6. If there are no available pods, then you need a larger instance type.
+