AWS eks documentation change
Summary
Updated instance recommendations, simplified technical details, added credential verification steps, and refined explanations for components like EKS Pod Identity Agent, Node Monitoring Agent, and Bottlerocket AMI.
Security assessment
The change adds a 'Check your credentials' section with AWS CLI validation steps, which promotes security best practices by ensuring proper authentication. However, other changes (e.g., removing mentions of 'CIS benchmark compliance' in Bottlerocket AMI) reduce security documentation without addressing vulnerabilities. The credential check addition is a security-conscious improvement but does not directly resolve a security issue.
Diff
diff --git a/eks/latest/userguide/ml-realtime-inference-cluster.md b/eks/latest/userguide/ml-realtime-inference-cluster.md index 6b66a04eb..5b39b40bc 100644 --- a//eks/latest/userguide/ml-realtime-inference-cluster.md +++ b//eks/latest/userguide/ml-realtime-inference-cluster.md @@ -55 +55 @@ Real-time online inference systems require a high-performance, resilient archite - * [Amazon G5 EC2 Instances](https://aws.amazon.com/ec2/instance-types/g5/) — For real-time inference workloads that demand GPU performance and cost efficiency, we are using g5.xlarge and g5.2xlarge G5 EC2 instance types, each featuring a single (1) NVIDIA A10G GPU with 24GB of memory (e.g., 8 billion parameters at FP16). They offer up to 3x higher performance and 40% better price-performance compared to previous generations like G4dn. Based on the NVIDIA Ampere Architecture, these GPUs are powered by NVIDIA A10G Tensor Core GPUs and 2nd generation AMD EPYC processors, support 4-8 vCPUs, up to 10 Gbps network bandwidth, and 250-450 GB of local NVMe SSD storage, ensuring fast data movement and compute power for complex models, making them ideal for low-latency, high-throughput inference tasks. Choosing an EC2 instance type is application-specific, depends on your model (e.g., image, video, text model), and your latency and throughput requirements for the inference workload. For instance, if using an image and or video model, you may want to use [P5 EC2 instances](https://aws.amazon.com/ec2/instance-types/p5/) for optimal, real-time latency. We recommend starting out with [G5 EC2 instances](https://aws.amazon.com/ec2/instance-types/g5/) as it provides a good starting point for getting up and running quickly, and then determining whether it’s the right fit for your workloads through performance benchmark testing. For more advanced cases, consider [G6 EC2 instances](https://aws.amazon.com/ec2/instance-types/g6/). + * [Amazon G5 EC2 Instances](https://aws.amazon.com/ec2/instance-types/g5/) — For GPU-intensive inference tasks, we are using g5.xlarge and g5.2xlarge G5 EC2 instance types, which feature a single (1) NVIDIA A10G GPU with 24GB of memory (e.g., 8 billion parameters at FP16). Based on the NVIDIA Ampere Architecture, these GPUs are powered by NVIDIA A10G Tensor Core GPUs and 2nd generation AMD EPYC processors, support 4-8 vCPUs, up to 10 Gbps network bandwidth, and 250-450 GB of local NVMe SSD storage, ensuring fast data movement and compute power for complex models, making them ideal for low-latency, high-throughput inference tasks. Choosing an EC2 instance type is application-specific, depends on your model (e.g., image, video, text model), and your latency and throughput requirements. For instance, if using an image and or video model, you may want to use [P5 EC2 instances](https://aws.amazon.com/ec2/instance-types/p5/) for optimal, real-time latency. We recommend starting out with [G5 EC2 instances](https://aws.amazon.com/ec2/instance-types/g5/) as it provides a good starting point for getting up and running quickly, then evaluating whether it’s the right fit for your workloads through performance benchmark testing. For more advanced cases, consider [G6 EC2 instances](https://aws.amazon.com/ec2/instance-types/g6/). @@ -57 +57 @@ Real-time online inference systems require a high-performance, resilient archite - * [Amazon EC2 M7g Instances](https://aws.amazon.com/ec2/instance-types/m7g/) — For CPU-intensive tasks like data preprocessing, API request handling, and non-GPU inference models, hosting the Karpenter controller, our add-ons, and other system components, we are using the m5.xlarge M7g EC2 instance type. M7g instances are ARM-based instance which features 4 vCPUs, 16 GB of memory, up to 12.5 Gbps network bandwidth, and is powered by AWS Graviton3 processors. They offer up to 25% better performance and higher memory bandwidth with DDR5 compared to previous Graviton2-based M6g instances, ensuring efficient compute power for lightweight, CPU-bound workloads. Choosing an EC2 instance type is application-specific and depends on your workload’s compute, memory, and scalability requirements. For compute-optimized workloads, you might consider [C7g EC2 instances](https://aws.amazon.com/ec2/instance-types/c7g/), which also use Graviton3 processors but are optimized for higher compute performance than M7g instances for certain use cases. Alternatively, newer [C8g EC2 instances](https://aws.amazon.com/ec2/instance-types/c8g/) (where available) provide up to 30% better compute performance than C7g instances. We recommend starting with M7g EC2 instances for their cost efficiency and compatibility with a wide range of workloads (e.g., application servers, microservices, gaming servers, mid-size data stores), then evaluating their suitability through performance benchmark testing. + * [Amazon EC2 M7g Instances](https://aws.amazon.com/ec2/instance-types/m7g/) — For CPU-intensive tasks like data preprocessing, API request handling, hosting the Karpenter controller, add-ons, and other system components, we are using the m5.xlarge M7g EC2 instance type. M7g instances are ARM-based instance which features 4 vCPUs, 16 GB of memory, up to 12.5 Gbps network bandwidth, and is powered by AWS Graviton3 processors. Choosing an EC2 instance type is application-specific and depends on your workload’s compute, memory, and scalability requirements. For compute-optimized workloads, you might consider [C7g EC2 instances](https://aws.amazon.com/ec2/instance-types/c7g/), which also use Graviton3 processors but are optimized for higher compute performance than M7g instances for certain use cases. Alternatively, newer [C8g EC2 instances](https://aws.amazon.com/ec2/instance-types/c8g/) (where available) provide up to 30% better compute performance than C7g instances. We recommend starting out with M7g EC2 instances for their cost efficiency and compatibility with a wide range of workloads (e.g., application servers, microservices, gaming servers, mid-size data stores), then evaluating whether it’s the right fit for your workloads through performance benchmark testing. @@ -59 +59 @@ Real-time online inference systems require a high-performance, resilient archite - * [Amazon S3 Mountpoint CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html) — For workloads on single-GPU instances where multiple pods share a GPU (e.g., multiple pods scheduled on the same node to utilize its GPU resources), we are using the Mountpoint S3 CSI Driver. It is especially useful when memory efficiency is critical, such as inference with large models in cost-sensitive, low-complexity setups. It exposes Amazon S3 buckets as a POSIX-like file system available to the Kubernetes cluster, which allows inference pods to read model artifacts (e.g., model weights) directly into memory without having to download them first and input datasets using standard file operations. Optimized for high-throughput, sequential read and write-heavy workloads, S3 has virtually unlimited storage capacity and accelerates data-intensive inference workloads. Choosing a storage CSI driver is application-specific, and depends on your workload’s throughput and latency requirements. Though the [FSx for OpenZFS CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/fsx-openzfs-csi.html) offers sub-millisecond latency for random I/O or fully POSIX-compliant shared persistent volumes across nodes, we recommend starting out with the Mountpoint S3 CSI Driver due to its scalability, lower costs for large datasets, and built-in integration with S3-managed object storage for read-heavy inference patterns (e.g., streaming model inputs), then determining whether it’s the right fit through performance benchmark testing. + * [Amazon S3 Mountpoint CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html) — For workloads on single-GPU instances where multiple pods share a GPU (e.g., multiple pods scheduled on the same node to utilize its GPU resources), we are using the Mountpoint S3 CSI Driver to optimize memory usage—essential for tasks like large-model inference in cost-sensitive, low-complexity setups. It exposes Amazon S3 buckets as a POSIX-like file system available to the Kubernetes cluster, which allows inference pods to read model artifacts (e.g., model weights) directly into memory without having to download them first, and input datasets using standard file operations. Additionally, S3 has virtually unlimited storage capacity and accelerates data-intensive inference workloads. Choosing a storage CSI driver is application-specific, and depends on your workload’s throughput and latency requirements. Though the [FSx for OpenZFS CSI Driver](https://docs.aws.amazon.com/eks/latest/userguide/fsx-openzfs-csi.html) offers sub-millisecond latency for random I/O or fully POSIX-compliant shared persistent volumes across nodes, we recommend starting out with the Mountpoint S3 CSI Driver due to its scalability, lower costs for large datasets, and built-in integration with S3-managed object storage for read-heavy inference patterns (e.g., streaming model inputs), then evaluating whether it’s the right fit for your workloads through performance benchmark testing. @@ -61 +61 @@ Real-time online inference systems require a high-performance, resilient archite - * [EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) — To enable secure and fine-grained access to AWS services, we are using the EKS Pod Identity Agent, which uses a single service principal and facilitates pod-level IAM role associations within the Amazon EKS cluster. EKS Pod Identity is EKS' modern method for securely granting IAM permissions to Kubernetes workloads, offering a streamlined alternative to the traditional [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) approach by utilizing a single service principal (pods.eks.amazonaws.com) instead of relying on individual OIDC providers for each cluster. This simplifies the configuration process by removing the need for complex trust policy setups like with IRSA, making it easier to assign permissions. Additionally, it enables roles to be reused across multiple clusters, promoting consistency and reducing administrative overhead. This not only reduces operational complexity but it supports advanced features like [IAM role session tags](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-abac.html) and [Target IAM roles](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-assign-target-role.html). + * [EKS Pod Identity Agent](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) — To enable access to AWS services, we are using the EKS Pod Identity Agent, which uses a single service principal and facilitates pod-level IAM role associations within the Amazon EKS cluster. EKS Pod Identity offers a streamlined alternative to the traditional [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) approach by utilizing a single service principal (pods.eks.amazonaws.com) instead of relying on individual OIDC providers for each cluster, which makes it easier to assign permissions. Additionally, it enables roles to be reused across multiple clusters and it supports advanced features like [IAM role session tags](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-abac.html) and [Target IAM roles](https://docs.aws.amazon.com/eks/latest/userguide/pod-id-assign-target-role.html). @@ -63 +63 @@ Real-time online inference systems require a high-performance, resilient archite - * [EKS Node Monitoring Agent](https://docs.aws.amazon.com/eks/latest/userguide/node-health.html) — To ensure continuous availability and reliability of inference services, we are using the EKS Node Monitoring Agent with Auto Repair, which automatically detects and replaces unhealthy nodes, minimizing downtime for real-time inference applications. It continuously monitors nodes for hardware, kernel, networking, and storage issues using enhanced health checks (e.g., KernelReady, NetworkingReady), and for GPU nodes, detects accelerator-specific failures, initiating graceful remediation by cordoning unhealthy nodes, waiting 10 minutes for transient GPU issues to resolve, and replacing nodes after 30 minutes for persistent failures. + * [EKS Node Monitoring Agent](https://docs.aws.amazon.com/eks/latest/userguide/node-health.html) — To ensure continuous availability and reliability of inference services, we are using the EKS Node Monitoring Agent with Auto Repair, which automatically detects and replaces unhealthy nodes, minimizing downtime. It continuously monitors nodes for hardware, kernel, networking, and storage issues using enhanced health checks (e.g., KernelReady, NetworkingReady). For GPU nodes, it detects accelerator-specific failures, initiating graceful remediation by cordoning unhealthy nodes, waiting 10 minutes for transient GPU issues to resolve, and replacing nodes after 30 minutes for persistent failures. @@ -65 +65 @@ Real-time online inference systems require a high-performance, resilient archite - * [Bottlerocket AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami-bottlerocket.html) — To provide a security-hardened and high-performance foundation for our EKS cluster, we are using the Bottlerocket AMI, which includes only the essential components required to run containers, minimizes attack surfaces through its immutable design, delivers superior security hardening with CIS benchmark compliance, and minimal boot times for fast scaling. The Bottlerocket AMI has a small operating system footprint and excels in running and orchestrating containers, making it a good choice for container-native environments. Choosing a node AMI is application-specific and depends on your workload’s customization, security, and scalability requirements. Though the [AL2023 AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) provides greater flexibility for host-level installations and customizations (e.g., specifying a dedicated cache directory in a PV/PVC without any additional node configurations), it can increase maintenance overhead compared to Bottlerocket’s container-native design. We recommend starting out with the Bottlerocket AMI for its smaller footprint and built-in optimization for containerized workloads (e.g., microservices, inference servers, scalable APIs), then validating through performance benchmark testing whether it’s the right fit for your workloads. + * [Bottlerocket AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami-bottlerocket.html) — To provide a security-hardened foundation for our EKS cluster, we are using the Bottlerocket AMI, which includes only the essential components required to run containers and offers minimal boot times for fast scaling. Choosing a node AMI is application-specific and depends on your workload’s customization, security, and scalability requirements. Though the [AL2023 AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html) provides greater flexibility for host-level installations and customizations (e.g., specifying a dedicated cache directory in a PV/PVC without any additional node configurations), we recommend starting out with the Bottlerocket AMI for its smaller footprint and built-in optimization for containerized workloads (e.g., microservices, inference servers, scalable APIs), then evaluating whether it’s the right fit for your workloads through performance benchmark testing. @@ -67 +67 @@ Real-time online inference systems require a high-performance, resilient archite - * [AWS Load Balancer Controller (LBC)](https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html) — To expose real-time inference endpoints with high availability and low-latency traffic routing, we are using the AWS Load Balancer Controller, which automatically provisions and manages Application Load Balancers (ALBs) for HTTP/HTTPS traffic and Network Load Balancers (NLBs) for TCP/UDP traffic based on Kubernetes Ingress and Service resources, enabling integration of inference models with external clients. It supports features like automatic target group binding, SSL termination, and path-based routing to distribute inference requests across multiple pods or nodes, ensuring scalability during traffic spikes while minimizing latency through AWS-native optimizations like connection multiplexing and health checks. + * [AWS Load Balancer Controller (LBC)](https://docs.aws.amazon.com/eks/latest/userguide/lbc-helm.html) — To expose real-time inference endpoints, we are using the AWS Load Balancer Controller, which automatically provisions and manages Application Load Balancers (ALBs) for HTTP/HTTPS traffic and Network Load Balancers (NLBs) for TCP/UDP traffic based on Kubernetes Ingress and Service resources, enabling the integration of inference models with external clients. Additionally, it supports features like path-based routing to distribute inference requests across multiple pods or nodes, ensuring scalability during traffic spikes and minimizing latency through AWS-native optimizations like connection multiplexing and health checks. @@ -77,0 +78,9 @@ By default, `eksctl` will create a dedicated VPC for the cluster with a CIDR blo +### Check your credentials + +Check whether your AWS CLI credentials are valid and can authenticate with AWS services: + + + aws sts get-caller-identity + +If successful, the CLI will return details about your AWS identity (UserId, Account, and Arn). + @@ -80 +89 @@ By default, `eksctl` will create a dedicated VPC for the cluster with a CIDR blo -G5 instance types are not available in all regions. Check your nearest region: +G5 instance types are not available in all regions. Check your nearest region. For example: @@ -85 +94 @@ G5 instance types are not available in all regions. Check your nearest region: -If the output is successful, the G5 instance type is available in the region you specified. +If successful, the G5 instance type is available in the region you specified. @@ -87 +96 @@ If the output is successful, the G5 instance type is available in the region you -The Bottlerocket AMI is not available in all regions. Check by retrieving a Bottlerocket AMI ID for your nearest region: +The Bottlerocket AMI is not available in all regions. Check by retrieving a Bottlerocket AMI ID for your nearest region. For example: @@ -93 +102 @@ The Bottlerocket AMI is not available in all regions. Check by retrieving a Bott -If the output is successful, the Bottlerocket AMI is available in the region you specified. +If successful, the Bottlerocket AMI is available in the region you specified. @@ -127 +136 @@ Karpenter needs specific IAM roles and policies (e.g., Karpenter controller IAM -The AWS LBC needs permission to provision and manage AWS load balancers, such as creating ALBs for Ingress resources or NLBs for services of type `LoadBalancer`. We’ll specify this permissions policy during cluster creation. Below, we create the IAM policy. During cluster creation, we will create the service account with eksctl in the ClusterConfig. Create the LBC IAM policy: +The AWS LBC needs permission to provision and manage AWS load balancers, such as creating ALBs for Ingress resources or NLBs for services of type `LoadBalancer`. We’ll specify this permissions policy during cluster creation. During cluster creation, we will create the service account with eksctl in the ClusterConfig. Create the LBC IAM policy: @@ -134 +143 @@ The AWS LBC needs permission to provision and manage AWS load balancers, such as -When the Mountpoint S3 CSI Driver is installed, its DaemonSet pods are configured to use a service account for execution. The Mountpoint for Amazon S3 CSI driver needs permission to interact with the Amazon S3 bucket you create later in this guide. We’ll specify this permissions policy during cluster creation. Below, we create the IAM policy. During cluster creation, we will create the service account with eksctl in the ClusterConfig. Create the S3 IAM policy: +When the Mountpoint S3 CSI Driver is installed, its DaemonSet pods are configured to use a service account for execution. The Mountpoint for Mountpoint S3 CSI driver needs permission to interact with the Amazon S3 bucket you create later in this guide. We’ll specify this permissions policy during cluster creation. During cluster creation, we will create the service account with eksctl in the ClusterConfig. Create the S3 IAM policy: @@ -522 +531 @@ Check the pod’s logs: -You should see the following response: +The expected output should look like this: