AWS sagemaker documentation change
Summary
Removed detailed command guide content and replaced with redirect notice to Nova user guides
Security assessment
The change removes operational documentation but doesn't address security vulnerabilities or weaknesses. It's a content relocation/archiving change rather than security-related modification. No evidence of security fixes or vulnerability remediation in the diff.
Diff
diff --git a/sagemaker/latest/dg/nova-hp-essential-commands-guide.md b/sagemaker/latest/dg/nova-hp-essential-commands-guide.md index 8e906be31..a250a45ad 100644 --- a//sagemaker/latest/dg/nova-hp-essential-commands-guide.md +++ b//sagemaker/latest/dg/nova-hp-essential-commands-guide.md @@ -5,2 +4,0 @@ -Installing Recipe CLIConnecting to your clusterStarting a training jobChecking job statusMonitoring job logsListing active jobsCanceling a jobRunning an evaluation jobCommon issues - @@ -9,461 +7 @@ Installing Recipe CLIConnecting to your clusterStarting a training jobChecking j -Amazon Amazon SageMaker HyperPod provides extensive command-line functionality for managing training workflows. This guide covers essential commands for common operations, from connecting to your cluster to monitoring job progress. - -###### Prerequisites - -Before using these commands, ensure you have completed the following setup: - - * HyperPod cluster with RIG created (typically in us-east-1) - - * Output Amazon S3 bucket created for training artifacts - - * IAM roles configured with appropriate permissions - - * Training data uploaded in correct JSONL format - - * FSx for Lustre sync completed (verify in cluster logs on first job) - - - - -###### Topics - - * Installing Recipe CLI - - * Connecting to your cluster - - * Starting a training job - - * Checking job status - - * Monitoring job logs - - * Listing active jobs - - * Canceling a job - - * Running an evaluation job - - * Common issues - - - - -## Installing Recipe CLI - -Navigate to the root of your recipe repository before running the installation command. - -###### Use the SageMaker HyperPodrecipes repository if using Non Forge customization techniques, for Forge based customization refer to the forge specific recipe repository. - -Run the following commands to install the HyperPod CLI: - -###### Note - -Make sure you aren’t in an active conda / anaconda / miniconda environment or another virtual environment - -If you are, please exit the environment using: - - * `conda deactivate` for conda / anaconda / miniconda environments - - * `deactivate` for python virtual environments - - - - -If you are using a Non Forge customization technique, download the sagemaker-hyperpod-recipes as shown below: - - - git clone -b release_v2 https://github.com/aws/sagemaker-hyperpod-cli.git cd sagemaker-hyperpod-cli - pip install -e . - cd .. - root_dir=$(pwd) - export PYTHONPATH=${root_dir}/sagemaker-hyperpod-cli/src/hyperpod_cli/sagemaker_hyperpod_recipes/launcher/nemo/nemo_framework_launcher/launcher_scripts:$PYTHONPATH - curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 - chmod 700 get_helm.sh - ./get_helm.sh rm -f ./get_helm.sh - -If you are a**Forge Subscriber,** you should be downloading the recipes using below mentioned process. - - - mkdir NovaForgeHyperpodCLI - cd NovaForgeHyperpodCLI - aws s3 cp s3://nova-forge-c7363-206080352451-us-east-1/v1/ ./ --recursive - pip install -e . - - curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 - chmod 700 get_helm.sh - ./get_helm.sh rm -f ./get_helm.sh - -###### Tip - -To use a [new virtual environment](https://docs.python.org/3/library/venv.html) before running `pip install -e .`, run: - - * `python -m venv nova_forge` - - * `source nova_forge/bin/activate` - - * Your command line will now display (nova_forge) at the beginning of your prompt - - * This ensures there are no competing dependencies when using the CLI - - - - -**Purpose** : Why do we do `pip install -e .` ? - -This command installs the HyperPod CLI in editable mode, allowing you to use updated recipes without reinstalling each time. It also enables you to add new recipes that the CLI can automatically pick up. - -## Connecting to your cluster - -Connect the HyperPod CLI to your cluster before running any jobs: - - - export AWS_REGION=us-east-1 && hyperpod connect-cluster --cluster-name <your-cluster-name> --region us-east-1 - -###### Important - -This command creates a context file (`/tmp/hyperpod_context.json`) that subsequent commands require. If you see an error about this file not found, re-run the connect command. - -**Pro tip** : You can further configure your cluster to always use the `kubeflow` namespace by adding the `--namespace kubeflow` argument to your command as follows: - - - export AWS_REGION=us-east-1 && \ - hyperpod connect-cluster \ - --cluster-name <your-cluster-name> \ - --region us-east-1 \ - --namespace kubeflow - -This saves you the effort of adding the `-n kubeflow` in every command when interacting with your jobs. - -## Starting a training job - -###### Note - -If running PPO/RFT jobs, ensure you add label selector settings to `src/hyperpod_cli/sagemaker_hyperpod_recipes/recipes_collection/cluster/k8s.yaml` so that all pods are schedule on the same node. - - - label_selector: - required: - sagemaker.amazonaws.com/instance-group-name: - - <rig_group> - -Launch a training job using a recipe with optional parameter overrides: - - - hyperpod start-job -n kubeflow \ - --recipe fine-tuning/nova/nova_1_0/nova_micro/SFT/nova_micro_1_0_p5_p4d_gpu_lora_sft \ - --override-parameters '{ - "instance_type": "ml.p5.48xlarge", - "container": "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-fine-tune-repo:SM-HP-SFT-latest" - }' - -**Expected output** : - - - Final command: python3 <path_to_your_installation>/NovaForgeHyperpodCLI/src/hyperpod_cli/sagemaker_hyperpod_recipes/main.py recipes=fine-tuning/nova/nova_micro_p5_gpu_sft cluster_type=k8s cluster=k8s base_results_dir=/local/home/<username>/results cluster.pullPolicy="IfNotPresent" cluster.restartPolicy="OnFailure" cluster.namespace="kubeflow" container="708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-fine-tune-repo:HP-SFT-DATAMIX-latest" - - Prepared output directory at /local/home/<username>/results/<job-name>/k8s_templates - Found credentials in shared credentials file: ~/.aws/credentials - Helm script created at /local/home/<username>/results/<job-name>/<job-name>_launch.sh - Running Helm script: /local/home/<username>/results/<job-name>/<job-name>_launch.sh - - NAME: <job-name> - LAST DEPLOYED: Mon Sep 15 20:56:50 2025 - NAMESPACE: kubeflow - STATUS: deployed - REVISION: 1 - TEST SUITE: None - Launcher successfully generated: <path_to_your_installation>/NovaForgeHyperpodCLI/src/hyperpod_cli/sagemaker_hyperpod_recipes/launcher/nova/k8s_templates/SFT - - { - "Console URL": "https://us-east-1.console.aws.amazon.com/sagemaker/home?region=us-east-1#/cluster-management/<your-cluster-name>" - } - -## Checking job status - -Monitor your running jobs using kubectl: - - - kubectl get pods -o wide -w -n kubeflow | (head -n1 ; grep <your-job-name>) - -###### Understanding pod statuses - -The following table explains common pod statuses: - -Status | Description ----|--- -`Pending` | Pod accepted but not yet scheduled onto a node, or waiting for container images to be pulled -`Running` | Pod bound to a node with at least one container running or starting -`Succeeded` | All containers completed successfully and won't restart -`Failed` | All containers terminated with at least one ending in failure -`Unknown` | Pod state cannot be determined (usually due to node communication issues) -`CrashLoopBackOff` | Container repeatedly failing; Kubernetes backing off from restart attempts -`ImagePullBackOff` / `ErrImagePull` | Unable to pull container image from registry