AWS eks documentation change
Summary
Restructured documentation to focus on use cases and case studies, removed security advantages section, added detailed technical implementations for AI/ML workloads
Security assessment
The changes primarily reorganize content and add implementation details for AI/ML workloads. While a previous security section mentioning IAM roles and encryption was removed, there's no evidence this addresses a specific security vulnerability. The update focuses on technical capabilities rather than security features.
Diff
diff --git a/eks/latest/userguide/machine-learning-on-eks.md b/eks/latest/userguide/machine-learning-on-eks.md index 2ce3bc5f2..13243ca7e 100644 --- a/eks/latest/userguide/machine-learning-on-eks.md +++ b/eks/latest/userguide/machine-learning-on-eks.md @@ -5 +5 @@ -Advantages of Machine Learning on EKS and the AWS cloudWhy Choose Amazon EKS for AI/ML?Start using Machine Learning on EKS +Why Choose EKS for AI/ML?Key use casesCase studiesStart using Machine Learning on EKS @@ -13 +13 @@ To contribute to this user guide, choose the **Edit this page on GitHub** link t -Machine Learning (ML) is an area of Artificial Intelligence (AI) where machines process large amounts of data to look for patterns and make connections between the data. This can expose new relationships and help predict outcomes that might not have been apparent otherwise. +Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes platform that empowers organizations to deploy, manage, and scale AI and machine learning (ML) workloads with unparalleled flexibility and control. Built on the open source Kubernetes ecosystem, EKS lets you harness your existing Kubernetes expertise, while integrating seamlessly with open source tools and AWS services. @@ -15 +15 @@ Machine Learning (ML) is an area of Artificial Intelligence (AI) where machines -For large-scale ML projects, data centers must be able to store large amounts of data, process data quickly, and integrate data from many sources. The platforms running ML applications must be reliable and secure, but also offer resiliency to recover from data center outages and application failures. AWS Elastic Kubernetes Service (EKS), running in the AWS cloud, is particularly suited for ML workloads. +Whether you’re training large-scale models, running real-time online inference, or deploying generative AI applications, EKS delivers the performance, scalability, and cost efficiency your AI/ML projects demand. @@ -17 +17 @@ For large-scale ML projects, data centers must be able to store large amounts of -The primary goal of this section of the EKS User Guide is to help you put together the hardware and software component to build platforms to run Machine Learning workloads in an EKS cluster. We start by explaining the features and services available to you in EKS and the AWS cloud, then provide you with tutorials to help you work with ML platforms, frameworks, and models. +## Why Choose EKS for AI/ML? @@ -19 +19 @@ The primary goal of this section of the EKS User Guide is to help you put togeth -## Advantages of Machine Learning on EKS and the AWS cloud +EKS is a managed Kubernetes platform that helps you deploy and manage complex AI/ML workloads. Built on the open source Kubernetes ecosystem, it integrates with AWS services, providing the control and scalability needed for advanced projects. For teams new to AI/ML deployments, existing Kubernetes skills transfer directly, allowing efficient orchestration of multiple workloads. @@ -21 +21 @@ The primary goal of this section of the EKS User Guide is to help you put togeth -Amazon Elastic Kubernetes Service (EKS) is a powerful, managed Kubernetes platform that has become a cornerstone for deploying and managing AI/ML workloads in the cloud. With its ability to handle complex, resource-intensive tasks, Amazon EKS provides a scalable and flexible foundation for running AI/ML models, making it an ideal choice for organizations aiming to harness the full potential of machine learning. +EKS supports everything from operating system customizations to compute scaling, and its open source foundation promotes technological flexibility, preserving choice for future infrastructure decisions. The platform provides the performance and tuning options AI/ML workloads require, supporting features such as: @@ -23 +23 @@ Amazon Elastic Kubernetes Service (EKS) is a powerful, managed Kubernetes platfo -Key Advantages of AI/ML Platforms on Amazon EKS include: + * Full cluster control to fine-tune costs and configurations without hidden abstractions @@ -25 +25 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: - * **Scalability and Flexibility** Amazon EKS enables organizations to scale AI/ML workloads seamlessly. Whether you’re training large language models that require vast amounts of compute power or deploying inference pipelines that need to handle unpredictable traffic patterns, EKS scales up and down efficiently, optimizing resource use and cost. + * Sub-second latency for real-time inference workloads in production @@ -27 +27 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: - * **High Performance with GPUs and Neuron Instances** Amazon EKS supports a wide range of compute options, including GPUs and AWS} Neuron instances, which are essential for accelerating AI/ML workloads. This support allows for high-performance training and low-latency inference, ensuring that models run efficiently in production environments. + * Advanced customizations like multi-instance GPUs, multi-cloud strategies, and OS-level tuning @@ -29 +29 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: - * **Integration with AI/ML Tools** Amazon EKS integrates seamlessly with popular AI/ML tools and frameworks like TensorFlow, PyTorch, and Ray, providing a familiar and robust ecosystem for data scientists and engineers. These integrations enable users to leverage existing tools while benefiting from the scalability and management capabilities of Kubernetes. + * Ability to centralize workloads using EKS as a unified orchestrator across AI/ML pipelines @@ -31 +30,0 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: - * **Automation and Management** Kubernetes on Amazon EKS automates many of the operational tasks associated with managing AI/ML workloads. Features like automatic scaling, rolling updates, and self-healing ensure that your applications remain highly available and resilient, reducing the overhead of manual intervention. @@ -33 +31,0 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: - * **Security and Compliance** Running AI/ML workloads on Amazon EKS provides robust security features, including fine-grained IAM roles, encryption, and network policies, ensuring that sensitive data and models are protected. EKS also adheres to various compliance standards, making it suitable for enterprises with strict regulatory requirements. @@ -35,0 +34 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: +## Key use cases @@ -36,0 +36,30 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: +Amazon EKS provides a robust platform for a wide range of AI/ML workloads, supporting various technologies and deployment patterns: + + * **Real-time (online) inference:** EKS powers immediate predictions on incoming data, such as fraud detection, with sub-second latency using tools like [TorchServe](https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-torchserve.html), [Triton Inference Server](https://aws.amazon.com/blogs/containers/quora-3x-faster-machine-learning-25-lower-costs-with-nvidia-triton-on-amazon-eks/), and [KServe](https://kserve.github.io/website/0.8/get_started/first_isvc/) on Amazon EC2 [Inf1](https://aws.amazon.com/ec2/instance-types/inf1/) and [Inf2](https://aws.amazon.com/ec2/instance-types/inf2/) instances. These workloads benefit from dynamic scaling with [Karpenter](https://karpenter.sh/) and [KEDA](https://keda.sh/), while leveraging [Amazon EFS](https://aws.amazon.com/efs/) for model sharding across pods. [Amazon ECR Pull Through Cache (PTC)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/pull-through-cache-creating-rule.html) accelerates model updates, and [Bottlerocket](https://aws.amazon.com/bottlerocket/) data volumes with [Amazon EBS](https://docs.aws.amazon.com/ebs/latest/userguide/what-is-ebs.html)-optimized volumes ensure fast data access. + + * **General model training:** Organizations leverage EKS to train complex models on large datasets over extended periods using the [Kubeflow Training Operator (KRO)](https://www.kubeflow.org/docs/components/trainer/), [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), and [Torch Distributed Elastic](https://pytorch.org/docs/stable/distributed.elastic.html) on [Amazon EC2 P4d](https://aws.amazon.com/ec2/instance-types/p4/) and [Amazon EC2 Trn1](https://aws.amazon.com/ec2/instance-types/trn1/) instances. These workloads are supported by batch scheduling with tools like [Volcano](https://volcano.sh/en/#home_slider), [Yunikorn](https://yunikorn.apache.org/), and [Kueue](https://kueue.sigs.k8s.io/). [Amazon EFS](https://aws.amazon.com/efs/) enables sharing of model checkpoints, and [Amazon S3](https://aws.amazon.com/s3/) handles model import/export with lifecycle policies for version management. + + * **Retrieval augmented generation (RAG) pipelines:** EKS manages customer support chatbots and similar applications by integrating retrieval and generation processes. These workloads often use tools like [Argo Workflows](https://argoproj.github.io/workflows/) and [Kubeflow](https://www.kubeflow.org/) for orchestration, vector databases like [Pinecone](https://www.pinecone.io/blog/serverless/), [Weaviate](https://weaviate.io/), or [Amazon OpenSearch](https://aws.amazon.com/opensearch-service/), and expose applications to users via the [Application Load Balancer Controller (LBC)](./aws-load-balancer-controller.html). [NVIDIA NIM](https://docs.nvidia.com/nim/index.html) optimizes GPU utilization, while [Prometheus](./prometheus.html) and [Grafana](https://aws.amazon.com/grafana/) monitor resource usage. + + * **Generative AI model deployment:** Companies deploy real-time content creation services on EKS, such as text or image generation, using [Ray Serve](https://docs.ray.io/en/latest/serve/index.html), [vLLM](https://github.com/vllm-project/vllm), and [Triton Inference Server](https://aws.amazon.com/blogs/containers/quora-3x-faster-machine-learning-25-lower-costs-with-nvidia-triton-on-amazon-eks/) on Amazon [EC2 G5](https://aws.amazon.com/ec2/instance-types/g5/) and [Inferentia](https://aws.amazon.com/ai/machine-learning/inferentia/) accelerators. These deployments optimize performance and memory utilization for large-scale models. [JupyterHub](https://jupyter.org/hub) enables iterative development, [Gradio](https://www.gradio.app/) provides simple web interfaces, and the [S3 Mountpoint CSI Driver](./s3-csi.html) allows mounting S3 buckets as file systems for accessing large model files. + + * **Batch (offline) inference:** Organizations process large datasets efficiently through scheduled jobs with [AWS Batch](https://docs.aws.amazon.com/batch/latest/userguide/what-is-batch.html) or [Volcano](https://volcano.sh/en/docs/schduler_introduction/). These workloads often use [Inf1](https://aws.amazon.com/ec2/instance-types/inf1/) and [Inf2](https://aws.amazon.com/ec2/instance-types/inf2/) EC2 instances for AWS [Inferentia](https://aws.amazon.com/ai/machine-learning/inferentia/) chips, Amazon EC2 [G4dn](https://aws.amazon.com/ec2/instance-types/g4/) instances for NVIDIA T4 GPUs, or [c5](https://aws.amazon.com/ec2/instance-types/c5/) and [c6i](https://aws.amazon.com/ec2/instance-types/c6i) CPU instances, maximizing resource utilization during off-peak hours for analytics tasks. The [AWS Neuron SDK](https://aws.amazon.com/ai/machine-learning/neuron/) and NVIDIA GPU drivers optimize performance, while MIG/TS enables GPU sharing. Storage solutions include [Amazon S3](https://aws.amazon.com/s3/) and Amazon [EFS](https://aws.amazon.com/efs/) and [FSx for Lustre](https://aws.amazon.com/fsx/lustre/), with CSI drivers for various storage classes. Model management leverages tools like [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/), [Argo Workflows](https://argoproj.github.io/workflows/), and [Ray Cluster](https://docs.ray.io/en/latest/cluster/getting-started.html), while monitoring is handled by [Prometheus](./prometheus.html), [Grafana](https://aws.amazon.com/grafana/) and custom model monitoring tools. + + + + +## Case studies + +Customers choose Amazon EKS for various reasons, such as optimizing GPU usage or running real-time inference workloads with sub-second latency, as demonstrated in the following case studies. For a list of all case studies for Amazon EKS, see [AWS Customer Success Stories](https://aws.amazon.com/solutions/case-studies/browse-customer-success-stories/?refid=cr_card&customer-references-cards.sort-by=item.additionalFields.sortDate&customer-references-cards.sort-order=desc&awsf.customer-references-location=*all&awsf.customer-references-industry=*all&awsf.customer-references-use-case=*all&awsf.language=language%23english&awsf.customer-references-segment=*all&awsf.content-type=*all&awsf.customer-references-product=product%23eks&awsm.page-customer-references-cards=1). + + * [Unitary](https://aws.amazon.com/solutions/case-studies/unitary-eks-case-study/?did=cr_card&trk=cr_card) processes 26 million videos daily using AI for content moderation, requiring high-throughput, low-latency inference and have achieved an 80% reduction in container boot times, ensuring fast response to scaling events as traffic fluctuates. + + * [Miro](https://aws.amazon.com/solutions/case-studies/miro-eks-case-study/), the visual collaboration platform supporting 70 million users worldwide, reported an 80% reduction in compute costs compared to their previous self-managed Kubernetes clusters. + + * [Synthesia](https://aws.amazon.com/solutions/case-studies/synthesia-case-study/?did=cr_card&trk=cr_card), which offers generative AI video creation as a service for customers to create realistic videos from text prompts, achieved a 30x improvement in ML model training throughput. + + * [Harri](https://aws.amazon.com/solutions/case-studies/harri-eks-case-study/?did=cr_card&trk=cr_card), providing HR technology for the hospitality industry, achieved 90% faster scaling in response to spikes in demand and reduced its compute costs by 30% by migrating to [AWS Graviton processors](https://aws.amazon.com/ec2/graviton/). + + * [Ada Support](https://aws.amazon.com/solutions/case-studies/ada-support-eks-case-study/), an AI-powered customer service automation company, achieved a 15% reduction in compute costs alongside a 30% increase in compute efficiency. + + * [Snorkel AI](https://aws.amazon.com/blogs/startups/how-snorkel-ai-achieved-over-40-cost-savings-by-scaling-machine-learning-workloads-using-amazon-eks/), which equips enterprises to build and adapt foundation models and large language models, achieved over 40% cost savings by implementing intelligent scaling mechanisms for their GPU resources. @@ -38 +66,0 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: -## Why Choose Amazon EKS for AI/ML? @@ -40 +67,0 @@ Key Advantages of AI/ML Platforms on Amazon EKS include: -Amazon EKS offers a comprehensive, managed environment that simplifies the deployment of AI/ML models while providing the performance, scalability, and security needed for production workloads. With its ability to integrate with a variety of AI/ML tools and its support for advanced compute resources, EKS empowers organizations to accelerate their AI/ML initiatives and deliver innovative solutions at scale. @@ -42 +68,0 @@ Amazon EKS offers a comprehensive, managed environment that simplifies the deplo -By choosing Amazon EKS, you gain access to a robust infrastructure that can handle the complexities of modern AI/ML workloads, allowing you to focus on innovation and value creation rather than managing underlying systems. Whether you are deploying simple models or complex AI systems, Amazon EKS provides the tools and capabilities needed to succeed in a competitive and rapidly evolving field.