AWS eks: EKS ML Terraform guide: Grafana ALB CIDR warning and IAM notes
Summary
Restructures the EKS AI/ML Terraform setup guide, adding a variable table that flags the default `my_cidr` of 0.0.0.0/0 for the Grafana ALB and documents IAM/Pod Identity associations for monitoring and S3 access.
Security assessment
The added variable table explicitly calls out that `my_cidr` defaults to `0.0.0.0/0`, allowing any source to reach the Grafana ALB, and instructs users to override it. This is concrete security hardening guidance for network exposure. The change also documents IAM policies and EKS Pod Identity associations for Prometheus remote-write, Grafana query access, and the model-storage S3 role. No CVE or specific incident is referenced.
Evidence
+`my_cidr` | `0.0.0.0/0` | Source CIDR allowed to reach the Grafana ALB. Override this, as described in the following warning.
Diff
diff --git a/eks/latest/userguide/ml-cluster-setup-tf.md b/eks/latest/userguide/ml-cluster-setup-tf.md index 83c89db..b4af5f7 100644 --- a//eks/latest/userguide/ml-cluster-setup-tf.md +++ b//eks/latest/userguide/ml-cluster-setup-tf.md @@ -7 +7 @@ -PrerequisitesStep 1: Download and deploy the Terraform codeStep 2: Create dynamic GPU NodePoolStep 3: Test with a sample PodStep 4: Add reserved capacity to the NodePool (optional)MonitoringModel weights S3 bucketNext stepsCleanup +PrerequisitesUnderstand the Terraform codeStep 1: Deploy Terraform to create the EKS cluster, load balancing, monitoring, and S3 bucketStep 2: Verify the monitoring stackStep 3: Verify the model weights S3 bucketNext stepsCleanup @@ -19 +19 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster, GPU-enabled nodes with EKS Auto Mode or Karpenter, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. +This section walks you through the steps to create the infrastructure required to run training or inference workloads on Amazon EKS by using Terraform. The steps include creating an EKS cluster with EKS Auto Mode or Karpenter for node auto-scaling, load balancing for in-cluster endpoints, a monitoring stack with Prometheus and Grafana, and Amazon S3 storage for model weights. You create the GPU NodePool that provisions GPU-enabled nodes in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -23 +23 @@ See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest -**High-level architecture and workflow** +The following diagram shows the AWS high-level architecture for this section’s setup. Inside your AWS account, the EKS cluster uses a NodeClass and NodePool to provision an On-Demand Capacity Reservation (ODCR) G instance and a Spot or On-Demand G instance. In the monitoring stack, Prometheus remote writes metrics to Amazon Managed Service for Prometheus, and Grafana reads them. An Amazon S3 bucket stores model weights. The NodeClass and NodePool are shown for context; you create them in the [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html) section. @@ -25,3 +25 @@ See the documentation for [EKS Auto Mode](https://docs.aws.amazon.com/eks/latest - - -The diagram shows the AWS high-level architecture for this section’s setup. + @@ -42,0 +41,5 @@ The resources you create in this tutorial, including EKS clusters, GPU instances + * `git`, to clone the sample repository. + + * At least one unused VPC in your AWS Region’s quota. The Terraform code always creates its own VPC, and the default quota is 5 VPCs per Region. + + @@ -44,0 +48,14 @@ The resources you create in this tutorial, including EKS clusters, GPU instances +Check how many VPCs the Region already has against your quota: + + + aws ec2 describe-vpcs --region [.replaceable]`region-code` --query 'length(Vpcs)' + + aws service-quotas get-service-quota \ + --service-code vpc \ + --quota-code L-F678F1CE \ + --query 'Quota.Value' \ + --region [.replaceable]`region-code` + +###### Important + +If the Region is already at its VPC quota, the apply fails partway through with `VpcLimitExceeded` after it has already created IAM roles, KMS keys, the S3 bucket, and the AMP workspace. Run `terraform destroy` to remove those resources before you retry in another Region, or request a quota increase. For more information, see [Requesting a quota increase](https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html). @@ -54,2 +70,0 @@ Verify your tool versions: -## Step 1: Download and deploy the Terraform code - @@ -59 +74 @@ This walkthrough uses the Terraform code in the [sample-eks-docs](https://github - git clone [email protected]:aws-samples/sample-eks-docs.git + git clone https://github.com/aws-samples/sample-eks-docs.git @@ -62 +77 @@ This walkthrough uses the Terraform code in the [sample-eks-docs](https://github -The repository has the following structure under the `ai-ml/set-up-cluster/` directory you just changed into: +The repository has the following structure under the `ai-ml/set-up-cluster/` directory you changed into: @@ -68,0 +84 @@ The repository has the following structure under the `ai-ml/set-up-cluster/` dir + ├── README.md @@ -69,0 +86,2 @@ The repository has the following structure under the `ai-ml/set-up-cluster/` dir + │ ├── helm-values/ + │ └── nodepools/ @@ -70,0 +89,2 @@ The repository has the following structure under the `ai-ml/set-up-cluster/` dir + ├── helm-values/ + └── nodepools/ @@ -72 +92 @@ The repository has the following structure under the `ai-ml/set-up-cluster/` dir -The repository provides two deployment paths. Choose only one and use it throughout the guide. +## Understand the Terraform code @@ -74 +94 @@ The repository provides two deployment paths. Choose only one and use it through - * **EKS Auto Mode** (`terraform/auto-mode/`) — In addition to the core [networking, storage, and load balancing add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html#addon-consider-auto), EKS Auto Mode includes and manages the following capabilities for training and inference workloads: EKS node monitoring agent, automatic node repair, [SOCI](https://github.com/awslabs/soci-snapshotter) snapshotter for fast container pulls, and GPU readiness for the default NodeClass. The NVIDIA device plugin is included in the Bottlerocket accelerated AMI that EKS Auto Mode uses for GPU-enabled nodes. +The `terraform/auto-mode/` and `terraform/karpenter/` directories are each a self-contained root module. They create the same infrastructure, but differ in how much of it they manage explicitly: EKS Auto Mode provides node auto-scaling, the ALB controller, and GPU drivers as managed capabilities, whereas the Karpenter variant installs those components itself. @@ -76 +96 @@ The repository provides two deployment paths. Choose only one and use it through - * **Self-managed Karpenter** (`terraform/karpenter/`) — On an EKS cluster without EKS Auto Mode, the Terraform code installs and configures the components required for training and inference workloads. This includes networking add-ons (VPC CNI, CoreDNS, kube-proxy), Karpenter, the EKS node monitoring agent, the NVIDIA device plugin, and SOCI snapshotter for fast container pulls. +Each directory contains the following files. Review them before you apply, so you know what the module creates in your account. @@ -77,0 +98,11 @@ The repository provides two deployment paths. Choose only one and use it through +File | What it creates +---|--- +`main.tf`, `versions.tf`, `variables.tf`, `outputs.tf` | Provider configuration, input variables, and the outputs printed after an apply. +`vpc.tf` | The VPC, subnets, and subnet tags the ALB and EKS require. +`eks.tf` | The EKS cluster, its add-ons, and (on the Karpenter path) the `system` managed node group. +`monitoring.tf` | The Amazon Managed Service for Prometheus workspace, the kube-prometheus-stack Helm release, the NVIDIA DCGM Exporter, and the EKS Pod Identity associations they use. Chart values live in `helm-values/`. +`s3.tf` | The model weights S3 bucket, the `model-storage-sa` ServiceAccount, and its Pod Identity association. +`storage-class.tf` | The default `gp3` `StorageClass`. +`nodepools.tf` | GPU NodePools, selected by `var.nodepools`. On the Karpenter path, also the always-on `general-purpose` NodePool for non-GPU workloads. Manifests live in `nodepools/`. +`karpenter.tf`, `nvidia-device-plugin.tf`, `alb-controller.tf`, `fsx-subnet.tf` | Karpenter path only. EKS Auto Mode covers this functionality natively, so it has `alb-ingress-class.tf` in place of `alb-controller.tf` and no Karpenter or device plugin installs. +`efa-device-plugin.tf`, `mpi-operator.tf`, `fsx-lustre.tf` | Optional add-ons for distributed training, disabled by default. See the following table. @@ -78,0 +110 @@ The repository provides two deployment paths. Choose only one and use it through +### What a default apply creates @@ -79,0 +112,22 @@ The repository provides two deployment paths. Choose only one and use it through +Several components are controlled by input variables. With no `-var` overrides, `terraform apply` creates the cluster, the monitoring stack, and the S3 bucket, and does not create any GPU capacity: + +Variable | Default | Effect +---|---|--- +`nodepools` | `{}` | No GPU NodePools, so no GPU nodes and no GPU charges. You enable a strategy in [Create a GPU NodePool for AI/ML inference](./ml-inference-gpu-nodepool.html). +`enable_amazon_prometheus` | `true` | Creates the AMP workspace and wires Prometheus remote-write to it. +`enable_dcgm_exporter` | `true` | Installs the NVIDIA DCGM Exporter. Its Pods stay at zero until GPU nodes join the cluster. +`enable_efa` | `false` | Skips the EFA Kubernetes device plugin and the MPI Operator. +`enable_fsx` | `false` | Skips the FSx for Lustre file system, CSI driver, and static `PersistentVolume`. +`my_cidr` | `0.0.0.0/0` | Source CIDR allowed to reach the Grafana ALB. Override this, as described in the following warning. + +Other variables set versions and names, including `region` (`us-east-2`), `cluster_name` (`ai-eks-docs`), and `kubernetes_version`. For the full list with descriptions, see `variables.tf` in your chosen directory. For usage examples beyond this guide, see the [README](https://github.com/aws-samples/sample-eks-docs/blob/main/ai-ml/set-up-cluster/terraform/README.md) on GitHub in the `terraform/` directory. + +## Step 1: Deploy Terraform to create the EKS cluster, load balancing, monitoring, and S3 bucket + +A single `terraform apply` creates all of the infrastructure for this section: the EKS cluster, load balancing, the monitoring stack, and the Amazon S3 bucket for model weights. + +The repository provides two deployment paths. **Choose only one and use it throughout the guide**. + + * **EKS Auto Mode** (`terraform/auto-mode/`) — In addition to the core [networking, storage, and load balancing add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html#addon-consider-auto), EKS Auto Mode includes and manages the following capabilities for training and inference workloads: EKS node monitoring agent, automatic node repair, [SOCI](https://github.com/awslabs/soci-snapshotter) snapshotter for fast container pulls, and GPU readiness for the default NodeClass. The NVIDIA device plugin is included in the Bottlerocket accelerated AMI that EKS Auto Mode uses for GPU-enabled nodes. + + * **Self-managed Karpenter** (`terraform/karpenter/`) — On an EKS cluster without EKS Auto Mode, the Terraform code installs and configures the components required for training and inference workloads. This includes networking add-ons (VPC CNI, CoreDNS, kube-proxy), Karpenter, the EKS node monitoring agent, the NVIDIA device plugin, and SOCI snapshotter for fast container pulls. @@ -81 +134,0 @@ The repository provides two deployment paths. Choose only one and use it through -###### Important @@ -83 +135,0 @@ The repository provides two deployment paths. Choose only one and use it through -Pick either EKS Auto Mode or self-managed Karpenter and use it throughout the guide. Switching mid-stream requires destroying the cluster and starting over. @@ -85 +136,0 @@ Pick either EKS Auto Mode or self-managed Karpenter and use it throughout the gu -**EKS cluster options: EKS Auto Mode and self-managed Karpenter** @@ -99,2 +149,0 @@ Treat source-IP allowlisting as a minimum safeguard, not a complete one. Also ch -### Deploy the cluster - @@ -103 +152 @@ Change into the directory for your chosen path, initialize Terraform, and apply: -Both variants default to the `us-east-2` region. To deploy in a different region, add `-var "region=`region-code`"` to the `terraform apply` command in the following step, where `region-code` is the AWS Region you want to deploy in. +To deploy in a different Region, add `-var "region=`region-code`"` to the `terraform apply` command that follows, where `region-code` is the AWS Region you want to deploy in. @@ -105 +154 @@ Both variants default to the `us-east-2` region. To deploy in a different region -The Terraform code uses all available Availability Zones in the target region, excluding `use1-az3`, `usw1-az2`, and `cac1-az3` because [Amazon EKS does not support control plane placement in those zones](https://repost.aws/knowledge-center/eks-cluster-creation-errors). +The Terraform code uses all available Availability Zones in the target Region, excluding `use1-az3`, `usw1-az2`, and `cac1-az3` because [Amazon EKS does not support control plane placement in those zones](https://repost.aws/knowledge-center/eks-cluster-creation-errors). @@ -144,0 +194,2 @@ The `configure_kubectl` output is a ready-to-run command that points `kubectl` a +The resource count differs from the preceding example in Regions with a different number of Availability Zones, because the code creates a subnet, a route table, and their associations per zone. + @@ -220,171 +271 @@ Expected output: -## Step 2: Create dynamic GPU NodePool - -GPU NodePools are opt-in. By default, `terraform apply` creates the cluster and monitoring stack with no GPU capacity and no GPU billing. To provision GPU nodes, pass the `nodepools` variable with a strategy name. - -Enable the `spot-ondemand` strategy, which provisions G-family GPU instances with a generation greater than 4, using Spot capacity with On-Demand as a fallback: - - - terraform apply -var 'nodepools={"spot-ondemand"={}}' - -This command applies the NodePool and NodeClass templates from the `nodepools/spot-ondemand/` directory. Both paths use the same NodePool API, but they differ in the NodeClass the NodePool references. - -EKS Auto Mode - - -The NodePool references the managed `default` NodeClass, which already selects the Bottlerocket accelerated AMI, the NVIDIA drivers, the NVIDIA device plugin, and SOCI parallel pull. The `spot-ondemand` strategy ships no NodeClass of its own on this path. - -Validate the NodePool: - - - kubectl get nodepools,nodeclasses - -Expected output. The `gpu-inf` NodePool joins the built-in `general-purpose` and `system` NodePools, and all three reference the managed `default` NodeClass: - - - NAME NODECLASS NODES READY AGE - nodepool.karpenter.sh/general-purpose default 0 True 12m - nodepool.karpenter.sh/gpu-inf default 0 True 20s - nodepool.karpenter.sh/system default 1 True 12m - - NAME ROLE READY AGE - nodeclass.eks.amazonaws.com/default ai-eks-docs-eks-auto-... True 12m - -Self-managed Karpenter - - -Terraform applies a custom `gpu-inf` `EC2NodeClass` alongside the NodePool. The `EC2NodeClass` pins the EKS-optimized AL2023 AMI alias, enables SOCI through the `FastImagePull` feature gate, and sets `instanceStorePolicy: RAID0` to move the containerd image cache onto local NVMe. - -Validate the NodePool and EC2NodeClass: - - - kubectl get nodepools,ec2nodeclasses - -Expected output. The `gpu-inf` pair joins the `general-purpose` NodePool and EC2NodeClass that Terraform creates for non-GPU workloads: - - - NAME NODECLASS NODES READY AGE - nodepool.karpenter.sh/general-purpose general-purpose 1 True 14m - nodepool.karpenter.sh/gpu-inf gpu-inf 0 True 25s - - NAME READY AGE - ec2nodeclass.karpenter.k8s.aws/general-purpose True 14m - ec2nodeclass.karpenter.k8s.aws/gpu-inf True 25s - -Both paths show `0` nodes for `gpu-inf` until a GPU workload is scheduled. EKS Auto Mode and Karpenter only launch nodes when pending Pods require them. - -## Step 3: Test with a sample Pod - -Test your GPU NodePool setup with an `nvidia-smi` Pod: - - - cat << EOF | kubectl apply -f - - apiVersion: v1 - kind: Pod - metadata: - name: nvidia-smi - labels: - guide: ai-eks-docs - spec: - restartPolicy: OnFailure - tolerations: - - key: "nvidia.com/gpu" - operator: "Exists" - effect: "NoSchedule" - containers: - - name: nvidia-smi - image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal - command: ["nvidia-smi"] - resources: - limits: