AWS sagemaker documentation change
Summary
Added new troubleshooting sections for inference add-on installation failures related to missing dependencies (CSI drivers, cert-manager, ALB Controller, KEDA operator) and stuck CRDs during model deployment. Reorganized content structure with quick reference categorization.
Security assessment
The changes address operational deployment failures due to missing Kubernetes components and configuration issues. While components like cert-manager (TLS certificates) and IAM roles have security aspects, the documentation changes focus on resolving installation dependencies rather than addressing specific vulnerabilities or introducing new security controls. No CVEs, exploits, or security advisories are referenced.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-ts.md b/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-ts.md index 34e875d16..e9f51951f 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-ts.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-ts.md @@ -5 +5 @@ -Certificate download timeoutModel deployment stuck in pending stateModel deployment failed state troubleshootingChecking model deployment progressVPC ENI permission issueIAM trust relationship issueMissing NVIDIA GPU plugin errorInference operator fails to start +Inference add-on installation failed due to missing CSI driversInference Custom Resource Definitions are missing during model deploymentInference add-on installation failed due to missing cert-managerInference add-on installation failed due to missing ALB ControllerInference add-on installation failed due to missing KEDA operatorCertificate download timeoutModel deployment stuck in pending stateModel deployment failed state troubleshootingChecking model deployment progressVPC ENI permission issueIAM trust relationship issueMissing NVIDIA GPU plugin errorInference operator fails to start @@ -9 +9 @@ Certificate download timeoutModel deployment stuck in pending stateModel deploym -This troubleshooting guide addresses common issues that can occur during SageMaker AI HyperPod inference deployment and operation. These problems typically involve VPC networking configuration, IAM permissions, Kubernetes resource management, and operator connectivity issues that can prevent successful model deployment or cause deployments to fail or remain in pending states. +This troubleshooting guide addresses common issues that can occur during Amazon SageMaker HyperPod inference deployment and operation. These problems typically involve VPC networking configuration, IAM permissions, Kubernetes resource management, and operator connectivity issues that can prevent successful model deployment or cause deployments to fail or remain in pending states. @@ -12,0 +13,679 @@ This troubleshooting guide uses the following terminology: **Troubleshooting ste +**Quick reference: Find your issue** + +Use the following categories to quickly locate the troubleshooting section relevant to your problem: + + * **Add-on installation issues** \- See Inference add-on installation failed due to missing CSI drivers, Inference Custom Resource Definitions are missing during model deployment, Inference add-on installation failed due to missing cert-manager, Inference add-on installation failed due to missing ALB Controller, Inference add-on installation failed due to missing KEDA operator + + * **Permission and IAM issues** \- See VPC ENI permission issue, IAM trust relationship issue, Inference operator fails to start + + * **Deployment and resource issues** \- See Model deployment stuck in pending state, Model deployment failed state troubleshooting, Missing NVIDIA GPU plugin error + + * **Certificate and networking issues** \- See Certificate download timeout, Inference add-on installation failed due to missing cert-manager + + + + +## Inference add-on installation failed due to missing CSI drivers + +**Problem:** The inference operator add-on creation fails because required CSI driver dependencies are not installed on the EKS cluster. + +### Symptoms and diagnosis + +**Error messages:** + +The following errors appear in the add-on creation logs or inference operator logs: + + + S3 CSI driver not installed (missing CSIDriver s3.csi.aws.com). + Please install the required CSI driver and see the troubleshooting guide for more information. + + FSx CSI driver not installed (missing CSIDriver fsx.csi.aws.com). + Please install the required CSI driver and see the troubleshooting guide for more information. + +**Diagnostic steps:** + + 1. Check if CSI drivers are installed: + + # Check for S3 CSI driver + kubectl get csidriver s3.csi.aws.com + kubectl get pods -n kube-system | grep mountpoint + + # Check for FSx CSI driver + kubectl get csidriver fsx.csi.aws.com + kubectl get pods -n kube-system | grep fsx + + 2. Check EKS add-on status: + + # List all add-ons + aws eks list-addons --cluster-name $EKS_CLUSTER_NAME --region $REGION + + # Check specific CSI driver add-ons + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name aws-mountpoint-s3-csi-driver --region $REGION 2>/dev/null || echo "S3 CSI driver not installed" + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name aws-fsx-csi-driver --region $REGION 2>/dev/null || echo "FSx CSI driver not installed" + + 3. Check inference operator add-on status: + + aws eks describe-addon \ + --cluster-name $EKS_CLUSTER_NAME \ + --addon-name amazon-sagemaker-hyperpod-inference \ + --region $REGION \ + --query "addon.{Status:status,Health:health,Issues:issues}" \ + --output json + + + + +### Resolution + +**Step 1: Install missing S3 CSI driver** + + 1. Create IAM role for S3 CSI driver (if not already created): + + # Set up service account role ARN (from installation steps) + export S3_CSI_ROLE_ARN=$(aws iam get-role --role-name $S3_CSI_ROLE_NAME --query 'Role.Arn' --output text 2>/dev/null || echo "Role not found") + echo "S3 CSI Role ARN: $S3_CSI_ROLE_ARN" + + 2. Install S3 CSI driver add-on: + + aws eks create-addon \ + --cluster-name $EKS_CLUSTER_NAME \ + --addon-name aws-mountpoint-s3-csi-driver \ + --addon-version v1.14.1-eksbuild.1 \ + --service-account-role-arn $S3_CSI_ROLE_ARN \ + --region $REGION + + 3. Verify S3 CSI driver installation: + + # Wait for add-on to be active + aws eks wait addon-active --cluster-name $EKS_CLUSTER_NAME --addon-name aws-mountpoint-s3-csi-driver --region $REGION + + # Verify CSI driver is available + kubectl get csidriver s3.csi.aws.com + kubectl get pods -n kube-system | grep mountpoint + + + + +**Step 2: Install missing FSx CSI driver** + + 1. Create IAM role for FSx CSI driver (if not already created): + + # Set up service account role ARN (from installation steps) + export FSX_CSI_ROLE_ARN=$(aws iam get-role --role-name $FSX_CSI_ROLE_NAME --query 'Role.Arn' --output text 2>/dev/null || echo "Role not found") + echo "FSx CSI Role ARN: $FSX_CSI_ROLE_ARN" + + 2. Install FSx CSI driver add-on: + + aws eks create-addon \ + --cluster-name $EKS_CLUSTER_NAME \ + --addon-name aws-fsx-csi-driver \ + --addon-version v1.6.0-eksbuild.1 \ + --service-account-role-arn $FSX_CSI_ROLE_ARN \ + --region $REGION + + # Wait for add-on to be active + aws eks wait addon-active --cluster-name $EKS_CLUSTER_NAME --addon-name aws-fsx-csi-driver --region $REGION + + # Verify FSx CSI driver is running + kubectl get pods -n kube-system | grep fsx + + + + +### Verify All Dependencies + +After installing the missing dependencies, verify they are running correctly before retrying the inference operator installation: + + + # Check all required add-ons are active + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name aws-mountpoint-s3-csi-driver --region $REGION + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name aws-fsx-csi-driver --region $REGION + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name metrics-server --region $REGION + aws eks describe-addon --cluster-name $EKS_CLUSTER_NAME --addon-name cert-manager --region $REGION + + # Verify all pods are running + kubectl get pods -n kube-system | grep -E "(mountpoint|fsx|metrics-server)" + kubectl get pods -n cert-manager + +## Inference Custom Resource Definitions are missing during model deployment + +**Problem:** Custom Resource Definitions (CRDs) are missing when you attempt to create model deployments. This issue occurs when you previously installed and deleted the inference add-on without cleaning up model deployments that have finalizers. + +### Symptoms and diagnosis + +**Root cause:** + +If you delete the inference add-on without first removing all model deployments, custom resources with finalizers remain in the cluster. These finalizers must complete before you can delete the CRDs. The add-on deletion process doesn't wait for CRD deletion to complete, which causes the CRDs to remain in a terminating state and prevents new installations. + +**To diagnose this issue** + + 1. Check whether CRDs exist. + + kubectl get crd | grep inference.sagemaker.aws.amazon.com + + 2. Check for stuck custom resources. + + # Check for JumpStartModel resources + kubectl get jumpstartmodels -A + + # Check for InferenceEndpointConfig resources + kubectl get inferenceendpointconfigs -A + + 3. Inspect finalizers on stuck resources. + + # Example for a specific JumpStartModel + kubectl get jumpstartmodels <model-name> -n <namespace> -o jsonpath='{.metadata.finalizers}' + + # Example for a specific InferenceEndpointConfig + kubectl get inferenceendpointconfigs <config-name> -n <namespace> -o jsonpath='{.metadata.finalizers}' + + + + +### Resolution + +Manually remove the finalizers from all model deployments that weren't deleted when you removed the inference add-on. Complete the following steps for each stuck custom resource. + +**To remove finalizers from JumpStartModel resources** + + 1. List all JumpStartModel resources across all namespaces. + + kubectl get jumpstartmodels -A + + 2. For each JumpStartModel resource, remove the finalizers by patching the resource to set metadata.finalizers to an empty array. + + kubectl patch jumpstartmodels <model-name> -n <namespace> -p '{"metadata":{"finalizers":[]}}' --type=merge + +The following example shows how to patch a resource named kv-l1-only. + + kubectl patch jumpstartmodels kv-l1-only -n default -p '{"metadata":{"finalizers":[]}}' --type=merge