AWS sagemaker: HyperPod Inference release notes: v3.7/v2.1.0 add-on, gateway, logging, IAM guidance
Summary
Adds release notes for HyperPod Inference Amazon EKS v2.1.0-eksbuild.1 / Operator v3.7 (pod tolerations, priorityClassName, opt-in CloudWatch log publishing with configurable verbosity, LMCache MP mode, FSx CSI driver and cert-manager updates) and for v2.0.0-eksbuild.2 / Operator v3.6 (HyperPod Inference Gateway, gateway integration on CRDs, pod annotations, new add-on schema with IRSA role ARNs and TLS bucket). Also adds a Prerequisites IAM policy statement for sending operator logs to CloudWatch, renames existing release-note headings to include add-on versions, and corrects a Helm upgrade example from image.tag=v3.1 to v3.1.2.
Security assessment
The change is primarily feature/release documentation, not a fix for a specific vulnerability or incident. It does add security-relevant guidance: an opt-in (disabled-by-default) CloudWatch log-publishing path with a least-privilege IAM statement scoped to arn:aws:logs:*:*:log-group:/aws/hyperpod/*/inference-operator* (logs:CreateLogGroup/CreateLogStream/PutLogEvents), configurable log verbosity, and TLS/IRSA role configuration for the new Inference Gateway add-on schema. These are audit/logging and permission-scoping hardening details with moderate security impact.
Evidence
+ * **CloudWatch Log Publishing** – Publish operator logs to Amazon CloudWatch through an opt-in FluentBit sidecar. The feature is **disabled by default**. Enable it in your deployment configuration when you want centralized log aggregation in CloudWatch.
Diff
diff --git a/sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.md b/sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.md index 0ca312a3a..63b3040a7 100644 --- a//sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.md +++ b//sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.md @@ -7 +7 @@ -v3.5v3.4v3.3v3.2v3.1.2v3.1v3.0v2.3 +v2.1.0-eksbuild.1 / Operator v3.7v2.0.0-eksbuild.2 / Operator v3.6v1.6.0-eksbuild.1 / Operator v3.5v1.5.0-eksbuild.1 / Operator v3.4v1.4.0-eksbuild.1 / Operator v3.3v1.3.0-eksbuild.1 / Operator v3.2v1.2.0-eksbuild.1 / Operator v3.1.2Operator v3.1Operator v3.0Operator v2.3 @@ -15 +15,186 @@ For information about SageMaker HyperPod Inference capabilities and deployment o -## SageMaker HyperPod Inference release notes: v3.5 +## SageMaker HyperPod Inference release notes: HyperPod Inference Amazon EKS v2.1.0-eksbuild.1 and Inference Operator v3.7 + +**Release Date:** September 21, 2026 + +**Summary** + +Amazon SageMaker HyperPod Inference Operator v3.7 adds pod tolerations and `priorityClassName` support for finer scheduling control, opt-in CloudWatch log publishing with configurable verbosity, and a MultiProcess (MP) mode for LMCache. It also updates the bundled Amazon FSx CSI driver. + +Amazon SageMaker HyperPod Inference Operator v3.7 is available in all AWS Regions where SageMaker HyperPod is supported. + +**Inference Operator** + +Inference Operator v3.7 includes the following features: + + * **Pod Tolerations** – Add the new `tolerations` field to your `InferenceEndpointConfig` or `JumpStartModel` to schedule inference pods onto tainted nodes, such as dedicated HyperPod GPU node pools. Set the field under `spec.kubernetes.tolerations` for `InferenceEndpointConfig`, or `spec.tolerations` for `JumpStartModel`. The field accepts the standard Kubernetes toleration structure. The operator propagates your tolerations to the managed deployment pods so they can run on nodes with matching taints. + + * **PriorityClass Configuration** – Set a Kubernetes `priorityClassName` on the inference operator, ALB controller, KEDA, and Inference Gateway pods through the EKS add-on `configurationValues` or the operator Helm chart. Use this to control scheduling priority and preemption order for these pods relative to other workloads on the cluster. + + * **CloudWatch Log Publishing** – Publish operator logs to Amazon CloudWatch through an opt-in FluentBit sidecar. The feature is **disabled by default**. Enable it in your deployment configuration when you want centralized log aggregation in CloudWatch. + + * **Configurable Log Verbosity** – Control console and CloudWatch log verbosity with new customer-settable knobs. Raise verbosity when troubleshooting a deployment, or lower it to reduce log volume and cost in steady state. Valid log levels are `debug`, `info`, `warn`, and `error`. + + * **LMCache MultiProcess (MP) Mode** – Enable MultiProcess mode for LMCache by setting `kvCacheSpec.mode`. MP mode runs a sidecar process that shares GPU memory through CUDA-IPC caching. This lets you offload KV cache data in multiple supported formats. + + + + +**Dependency Updates** + + * **FSx CSI driver** – This release upgrades the bundled `aws-fsx-csi-driver` from 1.11.0 to 1.17.0 and realigns cert-manager to v1.18.2. + + + + +**Prerequisites** + +To use the CloudWatch log publishing feature, your Inference Operator execution role must allow writing to CloudWatch Logs. If your execution role doesn't already include these permissions, add the following statement to the role's IAM policy: + + + { + "Sid": "CustomerSideOperatorLogs", + "Effect": "Allow", + "Action": [ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ], + "Resource": "arn:aws:logs:*:*:log-group:/aws/hyperpod/*/inference-operator*" + } + +If you installed the Inference Operator as an EKS add-on, the managed execution-role policy already includes these permissions and no action is required. + +### Upgrade to v3.7 or v2.1.0-eksbuild.1 + +**EKS Add-on upgrade:** + +If you installed the Inference Operator as an EKS Add-on, upgrade to the latest version: + + + CLUSTER=EKS_CLUSTER_NAME + REGION=REGION + + aws eks update-addon \ + --cluster-name $CLUSTER \ + --addon-name amazon-sagemaker-hyperpod-inference \ + --addon-version v2.1.0-eksbuild.1 \ + --resolve-conflicts OVERWRITE \ + --region $REGION + +To enable CloudWatch logging and configure log levels, add a `logging` block to the configuration you pass to `--configuration-values`. This parameter replaces the entire configuration, so include your existing keys along with the new block: + + + "logging": { + "cloudWatch": { "enabled": true, "logLevel": "info" }, + "console": { "logLevel": "debug" } + } + +**Helm upgrade:** + +If you already have the Inference Operator installed by using Helm, use the following commands to upgrade: + + + helm get values -n kube-system hyperpod-inference-operator \ + > current-values.yaml + + cd sagemaker-hyperpod-cli/helm_chart/HyperPodHelmChart/\ + charts/inference-operator + + helm upgrade hyperpod-inference-operator . -n kube-system \ + -f current-values.yaml --set image.tag=v3.7 + + # Verification + kubectl get deployment hyperpod-inference-operator-controller-manager \ + -n hyperpod-inference-system \ + -o jsonpath='{.spec.template.spec.containers[0].image}' + +To enable CloudWatch logging and configure log levels, add the corresponding `--set` flags: + + + helm upgrade hyperpod-inference-operator . -n kube-system \ + -f current-values.yaml --set image.tag=v3.7 \ + --set logging.cloudWatch.enabled=true \ + --set logging.cloudWatch.logLevel=info \ + --set logging.console.logLevel=debug + +## SageMaker HyperPod Inference release notes: HyperPod Inference Amazon EKS v2.0.0-eksbuild.2 and Inference Operator v3.6 + +**Release Date:** September 10, 2026 + +**Summary** + +The `v2.0.0-eksbuild.2` release of the HyperPod Inference Amazon EKS add-on introduces the HyperPod Inference Gateway, a Kubernetes-native, LLM-aware routing layer that distributes inference traffic across model-serving pods using request-body content and GPU-aware endpoint selection. Both the Inference Operator and the Inference Gateway ship together as part of this add-on release. + +This is a major version update of the add-on, from `v1.6.0-eksbuild.1` to `v2.0.0-eksbuild.2`. The version update reflects the addition of the Inference Gateway, and the release remains backward compatible with the existing add-on features. If you do not need either component, you can disable it in the add-on configuration by setting `inferenceGateway.enabled` or `inferenceOperator.enabled` to `false`. + +**Inference Gateway** + + * **Inference Gateway** – Route inference traffic for multiple models through a single gateway endpoint using three routing layers: a Body-Based Router that reads the request `model` field, the gateway with HTTPRoute for header-based routing, and a GPU-aware Endpoint Picker. Configure the gateway with the new `InferenceGatewayConfig` CRD, including per-model schedulers, scoring weights, LoRA adapter support, and optional TLS termination. See [Inference Gateway for Amazon SageMaker HyperPod Inference](./sagemaker-hyperpod-model-deployment-inference-gateway.html). + + + + +**Inference Operator** + + * **Gateway integration on`InferenceEndpointConfig` and `JumpStartModel`** – Added an optional `inferenceGateway` field to both CRDs. Set `inferenceGateway.enabled` to `true` to opt a model into a shared gateway. The operator then adds a scheduler entry for that model to an `InferenceGatewayConfig`, creating the gateway if it does not already exist. Use `inferenceGateway.name` to select the gateway. Models that specify the same name in the same namespace share one gateway, and a name that is not yet in use creates a new one. When the field is empty, the operator generates a unique name so the model gets its own gateway. The operator mirrors the gateway's readiness in `status.inferenceGateway`. For more information, see [Integration with the HyperPod Inference Operator](./sagemaker-hyperpod-model-deployment-inference-gateway.html#sagemaker-hyperpod-model-deployment-inference-gateway-operator-integration). + + * **Custom pod annotations on`InferenceEndpointConfig`** – Added an optional `podTemplateAnnotations` field under `spec.kubernetes`. The operator propagates these annotations to the underlying pods, so that pod-scraping integrations such as metrics autodiscovery can read them. + + + + +### Upgrade to v2.0.0-eksbuild.2 or v3.6 + +**Configuration schema changes:** + +The add-on configuration schema changed in this release. `inferenceGateway` is newly added and configures the HyperPod Inference Gateway. The schema also adds `inferenceOperator`, which gives you the flexibility to turn the Inference Operator on or off independently. Both components are enabled by default, with `inferenceGateway.enabled` and `inferenceOperator.enabled` set to `true`. To disable either component, set its `enabled` field to `false` in `--configuration-values`. + +**EKS Add-on upgrade:** + +If you installed the Inference Operator as an EKS Add-on, upgrade to `v2.0.0-eksbuild.2` with the following command: + + + CLUSTER=EKS_CLUSTER_NAME + REGION=REGION + ACCOUNT=AWS_ACCOUNT_ID + + aws eks update-addon \ + --cluster-name $CLUSTER --region $REGION \ + --addon-name amazon-sagemaker-hyperpod-inference \ + --addon-version v2.0.0-eksbuild.2 \ + --resolve-conflicts OVERWRITE \ + --configuration-values '{ + "executionRoleArn": "arn:aws:iam::<ACCOUNT>:role/<EXEC_ROLE>", + "tlsCertificateS3Bucket": "<TLS_BUCKET>", + "inferenceOperator": { "enabled": true }, + "inferenceGateway": { + "enabled": true, + "serviceAccount": { "roleArn": "arn:aws:iam::<ACCOUNT>:role/<CERT_ISSUER_ROLE>" } + }, + "keda": { "enabled": true, "auth": { "aws": { "irsa": { "enabled": true, "roleArn": "arn:aws:iam::<ACCOUNT>:role/<KEDA_IRSA_ROLE>" } } } }, + "alb": { "enabled": true, "serviceAccount": { "create": true, "roleArn": "arn:aws:iam::<ACCOUNT>:role/<ALB_IRSA_ROLE>" } }, + "jumpstartGatedModelDownloadRoleArn": "arn:aws:iam::<ACCOUNT>:role/<JUMPSTART_ROLE>" + }' + +**Helm upgrade:** + +The Inference Gateway is available only through the Amazon EKS add-on. If you manage the Inference Operator with Helm and want to continue using operator-only features, upgrade your Helm release with the following commands. We recommend migrating to the add-on to get the full capabilities of both the Inference Operator and the Inference Gateway. + + + helm get values -n kube-system hyperpod-inference-operator \ + > current-values.yaml + + cd sagemaker-hyperpod-cli/helm_chart/HyperPodHelmChart/\ + charts/inference-operator + + helm upgrade hyperpod-inference-operator . -n kube-system \ + -f current-values.yaml --set image.tag=v3.6 + + # Verification + kubectl get deployment hyperpod-inference-operator-controller-manager \ + -n hyperpod-inference-system \ + -o jsonpath='{.spec.template.spec.containers[0].image}' + +For migration from Helm to the Amazon EKS add-on, see Helm to EKS Add-on Migration. + +## SageMaker HyperPod Inference release notes: HyperPod Inference Amazon EKS v1.6.0-eksbuild.1 and Inference Operator v3.5 @@ -40 +225 @@ The operator applies your rules in addition to the instance-type targeting it de -### Upgrade to v3.5 +### Upgrade to v3.5 or v1.6.0-eksbuild.1 @@ -76 +261 @@ If you installed the Inference Operator as an EKS Add-on, upgrade to the latest -## SageMaker HyperPod Inference release notes: v3.4