AWS nova documentation change
Summary
Updated SageMaker deployment guide to add a new 'inference components' deployment option (Option A) alongside the existing single model endpoint method (Option B). Restructured steps, added detailed code examples for creating endpoints with inference components, and updated invocation and cleanup procedures to support both deployment models.
Security assessment
The changes are purely functional and architectural, introducing a new deployment pattern (inference components) for better resource management and multi-model hosting. There is no mention of security vulnerabilities, patches, or security incidents. The changes include standard resource configuration and deployment steps without any security-specific content or references to security fixes.
Diff
diff --git a/nova/latest/nova2-userguide/nova-sagemaker-inference-getting-started.md b/nova/latest/nova2-userguide/nova-sagemaker-inference-getting-started.md index 4d14f92ba..bc16c5471 100644 --- a//nova/latest/nova2-userguide/nova-sagemaker-inference-getting-started.md +++ b//nova/latest/nova2-userguide/nova-sagemaker-inference-getting-started.md @@ -7 +7 @@ -PrerequisitesStep 1: Configure AWS credentialsStep 2: Create a SageMaker execution roleStep 3: Configure model parametersStep 4: Create SageMaker model and endpoint configurationStep 5: Deploy the endpointStep 6: Invoke the endpointStep 7: Clean up resources (Optional) +PrerequisitesStep 1: Configure AWS credentialsStep 2: Create a SageMaker execution roleStep 3: Configure model parametersStep 4: Create SageMaker resources and deploy the endpointStep 5: Invoke the endpointStep 6: Clean up resources (Optional) @@ -35,0 +36,4 @@ The following are prerequisites to deploy Amazon Nova models on SageMaker infere +###### Tip + +For a quick end-to-end deployment, you can run the [Custom Nova Model SageMaker Inference notebook](https://github.com/aws-samples/amazon-nova-samples/blob/main/customization/Nova_2.0/05_deployment/Custom-Nova-Model-SageMaker-Inference.ipynb) to deploy a customized Amazon Nova model on SageMaker inference in a single notebook. + @@ -300 +304,124 @@ The code automatically generates consistent names for AWS resources: -## Step 4: Create SageMaker model and endpoint configuration +## Step 4: Create SageMaker resources and deploy the endpoint + +SageMaker offers two approaches for deploying models to real-time endpoints. Choose the approach that fits your use case: + + * **Inference components** (Recommended): Deploys models as inference components on an endpoint. This approach enables you to host multiple models on a single endpoint, scale models independently, and optimize resource utilization. + + * **Single model endpoints** : Deploys a single model directly to an endpoint using a model object and endpoint configuration. This approach is simpler to set up and suitable for development, testing, or workloads that require only one model per endpoint. + + + + +### Option A: Creating with inference components + +With inference components, you first create an endpoint, then deploy your model as an inference component on that endpoint. This decouples the model from the endpoint infrastructure, giving you more flexibility. + +**Create the endpoint configuration** + +Create an endpoint configuration that defines the infrastructure without specifying a model. The instance type and count are managed at the endpoint level: + + + # Create Endpoint Configuration for inference components + INFERENCE_COMPONENT_NAME = MODEL_NAME + "-IC" + + try: + config_response = sagemaker.create_endpoint_config( + EndpointConfigName=ENDPOINT_CONFIG_NAME, + ProductionVariants=[ + { + 'VariantName': 'primary', + 'InstanceType': INSTANCE_TYPE, + 'InitialInstanceCount': 1, + 'RoutingConfig': { + 'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS' + } + } + ], + Tags=[ + { + 'Key': 'sagemaker:nova-inference-component', + 'Value': 'true' + } + ] + ) + print("Endpoint configuration created successfully!") + print(f"Config ARN: {config_response['EndpointConfigArn']}") + + except sagemaker.exceptions.ClientError as e: + print(f"Error creating endpoint configuration: {e}") + +**Create and deploy the endpoint** + + + import time + + try: + endpoint_response = sagemaker.create_endpoint( + EndpointName=ENDPOINT_NAME, + EndpointConfigName=ENDPOINT_CONFIG_NAME + ) + print("Endpoint creation initiated successfully!") + print(f"Endpoint ARN: {endpoint_response['EndpointArn']}") + except Exception as e: + print(f"Error creating endpoint: {e}") + + # Wait for endpoint to be InService + print("Waiting for endpoint to be InService...") + print("This typically takes 5-10 minutes...\n") + + while True: + try: + response = sagemaker.describe_endpoint(EndpointName=ENDPOINT_NAME) + status = response['EndpointStatus'] + + if status == 'Creating': + print(f"⏳ Status: {status} - Provisioning infrastructure...") + elif status == 'InService': + print(f"✅ Status: {status}") + print(f"\nEndpoint '{ENDPOINT_NAME}' is ready.") + break + elif status == 'Failed': + print(f"❌ Status: {status}") + print(f"Failure Reason: {response.get('FailureReason', 'Unknown')}") + break + else: + print(f"Status: {status}") + except Exception as e: + print(f"Error checking endpoint status: {e}") + break + + time.sleep(30) + +**Create the inference component** + +Once the endpoint is InService, deploy your Amazon Nova model as an inference component: + + + try: + ic_response = sagemaker.create_inference_component( + InferenceComponentName=INFERENCE_COMPONENT_NAME, + EndpointName=ENDPOINT_NAME, + VariantName='primary', + Specification={ + 'Container': { + 'Image': IMAGE, + 'ArtifactUrl': MODEL_S3_LOCATION, + 'Environment': environment + }, + 'ComputeResourceRequirements': { + 'NumberOfCpuCoresRequired': 15, + 'NumberOfAcceleratorDevicesRequired': 4, + 'MinMemoryRequiredInMb': 25000 + } + }, + RuntimeConfig={ + 'CopyCount': 1 + } + ) + print("Inference component creation initiated!") + print(f"Inference Component ARN: {ic_response['InferenceComponentArn']}") + + except sagemaker.exceptions.ClientError as e: + print(f"Error creating inference component: {e}") + +Key parameters: @@ -302 +429 @@ The code automatically generates consistent names for AWS resources: -In this step, you'll create two essential resources: a SageMaker model object that references your Amazon Nova model artifacts, and an endpoint configuration that defines how the model will be deployed. + * `InferenceComponentName`: Unique identifier for your inference component @@ -304 +431 @@ In this step, you'll create two essential resources: a SageMaker model object th -**SageMaker Model** : A model object that packages the inference container image, model artifacts location, and environment configuration. This is a reusable resource that can be deployed to multiple endpoints. + * `EndpointName`: The endpoint to deploy the component on @@ -306 +433,56 @@ In this step, you'll create two essential resources: a SageMaker model object th -**Endpoint Configuration** : Defines the infrastructure settings for deployment, including instance type, instance count, and model variants. This allows you to manage deployment settings separately from the model itself. + * `Image`: Docker container image URI for Amazon Nova inference + + * `ArtifactUrl`: Amazon S3 location of your model artifacts + + * `Environment`: Environment variables configured in Step 3 + + * `NumberOfCpuCoresRequired`: Number of CPU cores required per model copy + + * `NumberOfAcceleratorDevicesRequired`: Number of accelerator devices (GPUs) required per model copy + + * `MinMemoryRequiredInMb`: Minimum memory in MB required per model copy + + * `CopyCount`: Number of model copies to deploy + + + + +**Monitor inference component deployment** + + + # Wait for inference component to be InService + print("Waiting for inference component deployment...") + print("This typically takes 10-20 minutes as the model is loaded...\n") + + while True: + try: + ic_desc = sagemaker.describe_inference_component( + InferenceComponentName=INFERENCE_COMPONENT_NAME + ) + ic_status = ic_desc['InferenceComponentStatus'] + + if ic_status == 'Creating': + print(f"⏳ Status: {ic_status} - Loading model artifacts...") + elif ic_status == 'InService': + print(f"✅ Status: {ic_status}") + print(f"\nInference component '{INFERENCE_COMPONENT_NAME}' is ready!") + break + elif ic_status == 'Failed': + print(f"❌ Status: {ic_status}") + print(f"Failure Reason: {ic_desc.get('FailureReason', 'Unknown')}") + break + else: + print(f"Status: {ic_status}") + except Exception as e: + print(f"Error checking inference component status: {e}") + break + + time.sleep(30) + +###### Note + +When invoking the endpoint in Step 5, you must include the `InferenceComponentName` parameter in your invoke calls. See Step 5 for details. + +### Option B: Creating with single model endpoints