AWS sagemaker documentation change
Summary
Updated documentation for SageMaker HyperPod model deployment to use Llama 3.1 8B model instead of DeepSeek, changed instance types to ml.g5.24xlarge, switched to vLLM inference runtime, simplified environment variables, and updated YAML configurations.
Security assessment
The changes involve model updates, instance type changes, and runtime configuration adjustments without any mention of vulnerabilities, security patches, or security features. The environment variable VLLM_REQUEST_TIMEOUT added is a standard operational parameter unrelated to security.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-ftm.md b/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-ftm.md index 47ad013db..6ae3c77d5 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-ftm.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-deploy-ftm.md @@ -21 +21 @@ Before you begin, verify that you've: - * Installed [kubectl](https://kubernetes.io/docs/reference/kubectl/) utility and configured [jq](https://jqlang.org/) in your terminal. + * Installed [kubectl](https://kubernetes.io/docs/reference/kubectl/) utility in your terminal. @@ -43,2 +43,2 @@ Check with your cluster admin to ensure permissions are granted for this role or - # NOTE: For sample deployment, we use g5.8xlarge for deepseek-r1 1.5b model which has sufficient memory and GPU - instance_type="ml.g5.8xlarge" + # NOTE: For sample deployment, we use g5.24xlarge for Llama 3.1 8B model which has sufficient memory and GPU + instance_type="ml.g5.24xlarge" @@ -57 +57 @@ Using Amazon FSx as the model source - export SAGEMAKER_ENDPOINT_NAME="deepseek15b-fsx" + export SAGEMAKER_ENDPOINT_NAME="llama-fsx" @@ -63 +63 @@ Using Amazon FSx as the model source - 3. The following is an example yaml file for creating an endpoint with Amazon FSx and a DeepSeek model. + 3. The following is an example yaml file for creating an endpoint with Amazon FSx and a Llama model. @@ -74,2 +74,2 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - name: lmcache-test - namespace: inf-update + name: $SAGEMAKER_ENDPOINT_NAME + namespace: $CLUSTER_NAMESPACE @@ -84 +84 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - modelLocation: deepseek-1-5b + modelLocation: Llama-3.1-8B-Instruct @@ -87,12 +87 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - environmentVariables: - - name: HF_MODEL_ID - value: /opt/ml/model - - name: SAGEMAKER_PROGRAM - value: inference.py - - name: SAGEMAKER_SUBMIT_DIRECTORY - value: /opt/ml/model/code - - name: MODEL_CACHE_ROOT - value: /opt/ml/model - - name: SAGEMAKER_ENV - value: '1' - image: 763104351884.dkr.ecr.us-east-2.amazonaws.com/huggingface-pytorch-tgi-inference:2.4.0-tgi2.3.1-gpu-py311-cu124-ubuntu22.04-v2.0 + image: vllm/vllm-openai:v0.10.1 @@ -100 +89 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - containerPort: 8080 + containerPort: 8000 @@ -107,2 +96 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - nvidia.com/gpu: 1 - # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1 + nvidia.com/gpu: 4 @@ -112,2 +100,13 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - nvidia.com/gpu: 1 - # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1 + nvidia.com/gpu: 4 + args: + - "--model" + - "/opt/ml/model" + - "--port" + - "8000" + - "--tensor-parallel-size" + - "4" + - "--served-model-name" + - "Llama-3.1-8B-Instruct" + environmentVariables: + - name: VLLM_REQUEST_TIMEOUT + value: "600" @@ -121 +120 @@ Using Amazon S3 as the model source - export SAGEMAKER_ENDPOINT_NAME="deepseek15b-s3" + export SAGEMAKER_ENDPOINT_NAME="llama-s3" @@ -125 +124 @@ Using Amazon S3 as the model source - export S3_MODEL_LOCATION="deepseek-qwen-1-5b" + export S3_MODEL_LOCATION="<your-s3-bucket-name>" @@ -127 +126 @@ Using Amazon S3 as the model source - 3. The following is an example yaml file for creating an endpoint with Amazon S3 and a DeepSeek model. + 3. The following is an example yaml file for creating an endpoint with Amazon S3 and a Llama model using vLLM as the inference runtime. @@ -135 +134 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - apiVersion: inference.sagemaker.aws.amazon.com/v1alpha1 + apiVersion: inference.sagemaker.aws.amazon.com/v1 @@ -141,4 +140,4 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - modelName: deepseek15b - endpointName: $SAGEMAKER_ENDPOINT_NAME - instanceType: ml.g5.8xlarge - invocationEndpoint: invocations + modelName: Llama-3.1-8B-Instruct + instanceType: ml.g5.24xlarge + invocationEndpoint: v1/chat/completions + replicas: 2 @@ -150 +149 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - modelLocation: deepseek15b + modelLocation: Llama-3.1-8B-Instruct @@ -153,10 +152 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - resources: - limits: - nvidia.com/gpu: 1 - # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1 - requests: - nvidia.com/gpu: 1 - # For MIG-enabled instances, use: nvidia.com/mig-1g.10gb: 1 - cpu: 25600m - memory: 102Gi - image: 763104351884.dkr.ecr.us-east-2.amazonaws.com/djl-inference:0.32.0-lmi14.0.0-cu124 + image: vllm/vllm-openai:v0.10.1 @@ -169,86 +158,0 @@ For clusters with GPU partitioning enabled, replace `nvidia.com/gpu` with the ap - environmentVariables: - - name: PYTHONHASHSEED - value: "123" - - name: OPTION_ROLLING_BATCH - value: "vllm" - - name: SERVING_CHUNKED_READ_TIMEOUT - value: "480" - - name: DJL_OFFLINE - value: "true" - - name: NUM_SHARD - value: "1" - - name: SAGEMAKER_PROGRAM - value: "inference.py" - - name: SAGEMAKER_SUBMIT_DIRECTORY - value: "/opt/ml/model/code" - - name: MODEL_CACHE_ROOT - value: "/opt/ml/model" - - name: SAGEMAKER_MODEL_SERVER_WORKERS - value: "1" - - name: SAGEMAKER_MODEL_SERVER_TIMEOUT - value: "3600" - - name: OPTION_TRUST_REMOTE_CODE - value: "true" - - name: OPTION_ENABLE_REASONING - value: "true" - - name: OPTION_REASONING_PARSER - value: "deepseek_r1" - - name: SAGEMAKER_CONTAINER_LOG_LEVEL - value: "20" - - name: SAGEMAKER_ENV - value: "1" - - name: MODEL_SERVER_TYPE - value: "vllm" - - name: SESSION_KEY - value: "x-user-id" - EOF - -Using Amazon S3 as the model source - - - 1. Set up a SageMaker endpoint name. - - export SAGEMAKER_ENDPOINT_NAME="deepseek15b-s3" - - 2. Configure the Amazon S3 bucket location where the model is located. - - export S3_MODEL_LOCATION="deepseek-qwen-1-5b" - - 3. The following is an example yaml file for creating an endpoint with Amazon S3 and a DeepSeek model. - - cat <<EOF> deploy_s3_inference.yaml - --- - apiVersion: inference.sagemaker.aws.amazon.com/v1 - kind: InferenceEndpointConfig - metadata: - name: lmcache-test - namespace: inf-update - spec: - modelName: Llama-3.1-8B-Instruct - instanceType: ml.g5.24xlarge - invocationEndpoint: v1/chat/completions - replicas: 2 - modelSourceConfig: - modelSourceType: s3 - s3Storage: - bucketName: bugbash-ada-resources - region: us-west-2 - modelLocation: models/Llama-3.1-8B-Instruct - prefetchEnabled: false - kvCacheSpec: - enableL1Cache: true - # enableL2Cache: true - # l2CacheSpec: - # l2CacheBackend: redis/sagemaker - # l2CacheLocalUrl: redis://redis.redis-system.svc.cluster.local:6379 - intelligentRoutingSpec: - enabled: true - tlsConfig: - tlsCertificateOutputS3Uri: s3://sagemaker-lmcache-fceb9062-tls-6f6ee470 - metrics: - enabled: true - modelMetrics: - port: 8000 - loadBalancer: - healthCheckPath: /health - worker: @@ -257 +161 @@ Using Amazon S3 as the model source - nvidia.com/gpu: "4" + nvidia.com/gpu: 4 @@ -259,4 +163,3 @@ Using Amazon S3 as the model source - cpu: "6" - memory: 30Gi - nvidia.com/gpu: "4" - image: lmcache/vllm-openai:latest + cpu: 30000m + memory: 100Gi + nvidia.com/gpu: 4 @@ -263,0 +167 @@ Using Amazon S3 as the model source + - "--model" @@ -265,2 +169,2 @@ Using Amazon S3 as the model source