AWS eks: Restructure EKS ML cluster setup guide and monitoring SA config
Summary
Restructures the AI/ML cluster setup walkthrough into four steps (cluster, load balancing, monitoring, S3 bucket), moves all GPU NodePool/EC2NodeClass/ODCR content out to a separate page, bumps the cluster Kubernetes version from 1.36 to 1.37, removes the manually created Prometheus service account in favor of the chart creating service accounts, refreshes expected command outputs, and adds cluster-deletion verification and cleanup guidance. Mostly editorial wording, diagram, and formatting updates.
Security assessment
The only security-adjacent edit flips the kube-prometheus-stack values file so the chart creates its own service accounts (Prometheus no longer relies on a manually pre-created ServiceAccount), which touches the identity used for the EKS Pod Identity IAM binding. There is no referenced vulnerability, CVE, or CVE fix; the rest of the diff is doc restructuring, wording polish, a Kubernetes 1.37 version bump, and expected-output/cleanup updates.
Evidence
+ create: true
Diff
diff --git a/eks/latest/userguide/ml-cluster-setup-cli.md b/eks/latest/userguide/ml-cluster-setup-cli.md index b1cee2a..f96a3d3 100644 --- a//eks/latest/userguide/ml-cluster-setup-cli.md +++ b//eks/latest/userguide/ml-cluster-setup-cli.md @@ -7 +7 @@ -PrerequisitesSet environment variablesCreate cluster and GPU NodePoolStep 1: Create clusterStep 2: Create dynamic GPU NodePoolStep 3: Test with a sample PodStep 4: Add reserved capacity to the NodePool (optional)MonitoringModel weights S3 bucketNext stepsCleanup +PrerequisitesSet environment variablesStep 1: Create the clusterStep 2: Set up load balancingStep 3: Create the monitoring stackStep 4: Create S3 bucket for model weightsNext stepsCleanup @@ -19 +19 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS via CLI commands. The steps include creating an EKS cluster, GPU-enabled nodes with EKS Auto Mode or Karpenter, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. +This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using CLI commands. The steps include creating an EKS cluster with EKS Auto Mode or Karpenter for node auto-scaling, load balancing for in-cluster endpoints, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. You create the GPU NodePool that provisions GPU-enabled nodes in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -23 +23 @@ See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest -**High-level architecture and workflow** +The following diagram shows the AWS high-level architecture for this section’s setup. Inside your AWS account, the EKS cluster uses a NodeClass and NodePool to provision an On-Demand Capacity Reservation (ODCR) G instance and a Spot or On-Demand G instance. The NodeClass and NodePool are shown for context; you create them in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -25 +25 @@ See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest - +In the monitoring stack, Prometheus remote writes metrics to Amazon Managed Service for Prometheus, and Grafana reads them. An Amazon S3 bucket stores model weights. @@ -27 +27,20 @@ See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest -The diagram shows the AWS high-level architecture for this section’s setup. The numbered steps on the right indicate the order in which you complete the configuration in the steps below. +The numbered steps in the following diagram and its legend indicate the order in which you complete the configuration: + + 1. Create the EKS cluster + + 2. Set up load balancing + + 3. Create the monitoring stack + + 4. Create the S3 bucket + + + + + + +###### Note + +The components you install here that target GPU nodes are DaemonSets, so they report zero Pods until GPU nodes join the cluster. This includes the NVIDIA device plugin on the self-managed Karpenter path and the NVIDIA DCGM Exporter in the monitoring stack. After you create the GPU NodePool and a GPU node joins the cluster, Kubernetes schedules these DaemonSets onto it automatically. GPU metrics and the GPU dashboards in Grafana stay empty until then. + +Skip to [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) if you already have a cluster running with all of the necessary components. @@ -57 +76 @@ If you are on a version older than 0.227.0, follow the [eksctl installation guid -Keep the following cluster name and AWS Region consistent throughout these steps. Changing it may cause subsequent commands to target the wrong EKS cluster. +Keep the following cluster name and AWS Region consistent throughout these steps. Changing it might cause subsequent commands to target the wrong EKS cluster. @@ -81 +100 @@ Expected output: -The AZs in the output will vary per region. This example shows the available AZs for `us-east-2` region. +The AZs in the output vary per Region. This example shows the available AZs for the `us-east-2` Region. @@ -83 +102 @@ The AZs in the output will vary per region. This example shows the available AZs -## Create cluster and GPU NodePool +## Step 1: Create the cluster @@ -85 +104 @@ The AZs in the output will vary per region. This example shows the available AZs -This section provides two paths for creating your EKS cluster and GPU-enabled nodes, shown in the following diagram. Choose only one option throughout the guide. +This section provides two paths for creating your EKS cluster, shown in the following diagram. Choose only one option throughout the guide. @@ -94,2 +112,0 @@ This section provides two paths for creating your EKS cluster and GPU-enabled no -**EKS cluster options: EKS Auto Mode and self-managed Karpenter** - @@ -98,3 +115 @@ This section provides two paths for creating your EKS cluster and GPU-enabled no -In each of the following steps, choose a path (EKS Auto Mode, Karpenter) and follow it throughout. After completing the steps for your chosen path, you’ll have an EKS cluster with a GPU NodePool ready to schedule GPU workloads. - -## Step 1: Create cluster +In each of the following steps, choose a path (EKS Auto Mode, Karpenter) and follow it throughout. After completing the steps for your chosen path, you’ll have an EKS cluster ready for GPU workloads. You create the GPU NodePool that schedules GPU workloads in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -118 +133 @@ EKS Auto Mode - --version=1.36 \ + --version=1.37 \ @@ -167 +182 @@ Self-managed Karpenter -Public ECR is a global service hosted in `us-east-1`. Use `--region us-east-1` here regardless of which region your EKS cluster is in. +Public ECR is a global service hosted in `us-east-1`. Use `--region us-east-1` here regardless of which Region your EKS cluster is in. @@ -276 +291 @@ Wait for the Karpenter pods to roll out: -The EKS-optimized AL2023 AMI does not include the [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) (unlike the Bottlerocket AMI used by EKS Auto Mode). Install it via Helm to make GPU resources usable with Pods on the cluster. +The EKS-optimized AL2023 AMI does not include the [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) (unlike the Bottlerocket AMI used by EKS Auto Mode). Install it by using Helm to make GPU resources usable with Pods on the cluster. @@ -304 +319 @@ The EKS-optimized AL2023 AMI does not include the [NVIDIA device plugin](https:/ - * `gfd.enabled: true`: enables GPU Feature Discovery labels (`nvidia.com/gpu.product`, `nvidia.com/gpu.memory`, etc.) + * `gfd.enabled: true`: enables GPU Feature Discovery labels (`nvidia.com/gpu.product`, `nvidia.com/gpu.memory`, and others) @@ -320 +335 @@ Expected output: -Next, create a general-purpose NodePool. The `system` managed node group runs the add-ons and controllers. Add a general-purpose Karpenter `NodePool` so non-GPU workloads can scale on demand without landing on the `system` nodes. The GPU `NodePool` is created later in Step 2. +Next, create a general-purpose NodePool. The `system` managed node group runs the add-ons and controllers. Add a general-purpose Karpenter `NodePool`, so non-GPU workloads can scale on demand without landing on the `system` nodes. Creating this NodePool does not launch any nodes by itself: Karpenter provisions CPU nodes from it only when Pods need capacity that the `system` managed node group can’t provide. You create the GPU `NodePool` in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -372 +387 @@ Next, create a general-purpose NodePool. The `system` managed node group runs th -This NodePool has no taints, so any Pod that does not tolerate the GPU taint can schedule on it. +This NodePool has no taints, so any Pod that does not tolerate the GPU taint can schedule on it. It selects the `c`, `m`, and `r` instance families, which are CPU-only, so it never provisions GPU nodes. @@ -399 +414 @@ For both the EKS Auto Mode and self-managed Karpenter paths, automatic node repa -### Set up load balancing +## Step 2: Set up load balancing @@ -405,2 +419,0 @@ The ALB is placed in your VPC subnets based on AWS resource tags. Public subnets -#### Create the load balancer IngressClass - @@ -445 +458 @@ Create the IAM policy: -The cluster already installs the `eks-pod-identity-agent` add-on in Step 1, so EKS Pod Identity is available. Create a Pod Identity association that binds the IAM policy to the controller’s service account: +The cluster already installs the `eks-pod-identity-agent` add-on when you create the EKS cluster, so EKS Pod Identity is available. Create a Pod Identity association that binds the IAM policy to the controller’s service account: @@ -507,541 +520 @@ Both paths now have an `alb` `IngressClass`. The Grafana and inference sections -## Step 2: Create dynamic GPU NodePool - -Define a NodePool that dynamically provisions G-family GPU instances with a generation greater than 4, using Spot capacity with On-Demand as a fallback. The EKS Auto Mode and Karpenter paths both use the same NodePool API with the only difference being the NodeClass it points to. In EKS Auto Mode, the bundled `default` NodeClass already selects the right AMI and configures SOCI parallel pull, so the NodePool is the only object you create. In self-managed Karpenter, you also need a custom `EC2NodeClass` that pins the AMI and tunes SOCI. Because the two paths use different AMIs, the instance constraints differ between them, as noted in each path. - -EKS Auto Mode - - -###### Important - -This path uses the EKS-optimized Bottlerocket NVIDIA AMI, which does not include a driver version that supports the `g7` instance family (`g7` requires NVIDIA driver version 595 or later). Therefore, the NodePool in this path excludes the `g7` family. Other generation-7 instances, such as `g7e`, run on NVIDIA driver 580 and are supported. If you need `g7` support, use the self-managed Karpenter path with the EKS-optimized AL2023 NVIDIA AMI. For more information, see [EKS-optimized NVIDIA AMIs](./ml-eks-optimized-ami.html#eks-amis-nvidia). - -In EKS Auto Mode, the bundled `default` NodeClass automatically selects the Bottlerocket AMI for GPU instances, which includes pre-installed NVIDIA drivers, the NVIDIA device plugin, and SOCI parallel pull. You just need to apply a NodePool that references the `default` NodeClass: - - - cat << 'EOF' | kubectl apply -f - - apiVersion: karpenter.sh/v1 - kind: NodePool - metadata: - name: gpu-inf - spec: - template: - metadata: - labels: - guide: ai-eks-docs - spec: - nodeClassRef: - group: eks.amazonaws.com - kind: NodeClass - name: default - taints: - - key: nvidia.com/gpu - effect: NoSchedule - requirements: - - key: karpenter.sh/capacity-type - operator: In - values: ["spot", "on-demand"] - - key: eks.amazonaws.com/instance-category - operator: In - values: ["g"] - - key: eks.amazonaws.com/instance-generation - operator: Gt - values: ["4"] - - key: eks.amazonaws.com/instance-family - operator: NotIn - values: ["g7"] - - key: kubernetes.io/arch - operator: In - values: ["amd64"] - limits: - cpu: 1000 - memory: 5000Gi - EOF - -This NodePool provisions G-family GPU instances with a generation greater than 4, excluding the `g7` family (for example, [G5](https://aws.amazon.com/ec2/instance-types/g5/), [G6e](https://aws.amazon.com/ec2/instance-types/g6e/), and G7e). EKS Auto Mode chooses an instance type within these constraints. - -Self-managed Karpenter - - -Self-managed Karpenter does not include a default NodeClass. You first create an `EC2NodeClass` that pins the EKS-optimized NVIDIA AL2023 AMI alias, enables SOCI via the `FastImagePull` feature gate, and configures `instanceStorePolicy: RAID0` to move the containerd image cache to local NVMe. Then you create the NodePool that references it. - -**Create the EC2NodeClass** - -###### Example - - - cat << EOF | kubectl apply -f - - apiVersion: karpenter.k8s.aws/v1 - kind: EC2NodeClass - metadata: - name: gpu-inf - labels: - guide: ai-eks-docs - spec: - role: "eksctl-KarpenterNodeRole-${CLUSTER_NAME}" - amiSelectorTerms: - - alias: al2023@latest - subnetSelectorTerms: - - tags: - karpenter.sh/discovery: ${CLUSTER_NAME} - securityGroupSelectorTerms: - - tags: - karpenter.sh/discovery: ${CLUSTER_NAME} - tags: - karpenter.sh/discovery: ${CLUSTER_NAME} - instanceStorePolicy: RAID0 - userData: | - apiVersion: node.eks.aws/v1alpha1 - kind: NodeConfig - spec: - featureGates: - FastImagePull: true - EOF - -`instanceStorePolicy: RAID0` assembles local NVMe disks into a RAID-0 array. The `al2023@latest` AMI alias resolves to the EKS-optimized AL2023 AMI. When Karpenter launches a GPU instance type, it automatically selects the AL2023_x86_64_NVIDIA accelerated variant, which includes the NVIDIA driver pre-installed. - -The `FastImagePull` feature gate enables SOCI snapshotter parallel pull mode, which downloads and unpacks image layers concurrently. This matches the EKS Auto Mode behavior on G, P, and Trn instance families. - -For more SOCI tuning options (concurrent downloads per image, chunk size, etc.), see the [Karpenter SOCI blueprint](https://github.com/aws-samples/karpenter-blueprints/tree/main/blueprints/soci-snapshotter). - -Validate the EC2NodeClass: - - - kubectl get ec2nodeclass gpu-inf - -Expected output: `READY True`. If `False`, run `kubectl describe ec2nodeclass gpu-inf` and check conditions for missing subnet or security group tags. - -**Create the GPU NodePool** - -###### Example - - - cat << EOF | kubectl apply -f - - apiVersion: karpenter.sh/v1 - kind: NodePool