AWS sagemaker documentation change
Summary
Added troubleshooting section for add-on upgrade failure from v1.3.x to v1.5.0 with CRD migration instructions
Security assessment
The change addresses an operational issue with add-on upgrades and provides recovery steps, but there is no evidence of a security vulnerability being patched or disclosed.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md b/sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md index 5130dd664..e30c7ca6a 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-eks-operate-console-ui-governance-troubleshoot.md @@ -7 +7 @@ -Dashboard tabTasks tabPoliciesDeleting clustersUnallocated resource sharing +Dashboard tabTasks tabPoliciesDeleting clustersUnallocated resource sharingAdd-on upgrade from v1.3.x to v1.5.0 fails @@ -24,0 +25,2 @@ The following page contains known solutions for troubleshooting your HyperPod EK + * Add-on upgrade from v1.3.x to v1.5.0 fails + @@ -129,0 +132,28 @@ Verify MIG-enabled nodes show `success` state. +## Add-on upgrade from v1.3.x to v1.5.0 fails + +**Symptom:** When upgrading the task governance add-on directly from v1.3.x (Kueue v0.12) to v1.5.0 (Kueue v0.18) using `aws eks update-addon`, the add-on enters `UPDATE_FAILED` status with the following error: + + + CustomResourceDefinition.apiextensions.k8s.io "cohorts.kueue.x-k8s.io" is invalid: + status.storedVersions[0]: Invalid value: "v1alpha1": missing from spec.versions; + v1alpha1 was previously a storage version, and must remain in spec.versions until + a storage migration ensures no data remains persisted in v1alpha1 and removes + v1alpha1 from status.storedVersions + +**Resolution:** Use the upgrade option in the SageMaker AI HyperPod console. The console automatically handles the CRD migration by backing up existing resources, migrating storedVersions, upgrading the add-on, and restoring resources. + +###### Note + +If you already attempted a direct v1.3.x to v1.5.0 upgrade and it failed, the add-on will be in `UPDATE_FAILED` status with stale CRD definitions. To recover: + + 1. Delete the task governance add-on. + + 2. Delete all Kueue CRDs manually: + + kubectl get crds -o name | grep kueue | xargs kubectl delete + + 3. Reinstall the add-on at your target version using the SageMaker AI console or Amazon EKS console. + + + +