AWS sagemaker documentation change
Summary
Added new troubleshooting section 'Unallocated resource sharing' with diagnostic commands for HyperPod EKS clusters
Security assessment
The changes provide operational troubleshooting steps for resource allocation issues. There's no evidence of security vulnerability fixes, security incident references, or security controls implementation. The kubectl commands focus on resource monitoring and cluster health, not security configurations.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md b/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md index 28be90a8d..4839c3021 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md @@ -5 +5 @@ -Dashboard tabTasks tabPoliciesDeleting clusters +Dashboard tabTasks tabPoliciesDeleting clustersUnallocated resource sharing @@ -20,0 +21,2 @@ The following page contains known solutions for troubleshooting your HyperPod EK + * Unallocated resource sharing + @@ -86,0 +89,39 @@ The following lists known solutions to errors relating to deleting clusters. +## Unallocated resource sharing + +If your unallocated resource pool capacity is less than expected: + + 1. **Check node ready status** + + kubectl get nodes + +Verify all nodes show `Ready` status in the STATUS column. + + 2. **Check node schedulable status** + + kubectl get nodes -o custom-columns=NAME:.metadata.name,UNSCHEDULABLE:.spec.unschedulable + +Verify nodes show `<none>` or `false` (not `true`). + + 3. **List unallocated resource sharing ClusterQueues:** + + kubectl get clusterqueue | grep hyperpod-ns-idle-resource-sharing + +This shows all unallocated resource sharing ClusterQueues. If the ClusterQueues are not showing up, check the `FailureReason` under ClusterSchedulerConfig policy to see if there are any failure messages to continue the debugging. + + 4. **Verify unallocated resource sharing quota:** + + kubectl describe clusterqueue hyperpod-ns-idle-resource-sharing-<index> + +Check the `spec.resourceGroups[].flavors[].resources` section to see the quota allocated for each resource flavor. + +Multiple unallocated resource sharing ClusterQueues may exist depending on the number of resource flavors in your cluster. + + 5. **Check MIG configuration status (GPU nodes):** + + kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/mig\.config\.state}{"\n"}{end}' + +Verify MIG-enabled nodes show `success` state. + + + +