AWS eks documentation change
Summary
Restructured documentation with expanded guidance on container image optimization techniques including pre-caching strategies, SOCI snapshotter usage, image size reduction, and ML model artifact handling. Added implementation details and tradeoffs for different approaches.
Security assessment
Changes focus on performance optimization and operational efficiency improvements for container startup times. No security vulnerabilities, access controls, or encryption mechanisms are mentioned. The 'authentication' reference in model downloading is a generic mention without security implementation details.
Diff
diff --git a/eks/latest/best-practices/aiml-performance.md b/eks/latest/best-practices/aiml-performance.md index 9fe97dc55..802357c65 100644 --- a//eks/latest/best-practices/aiml-performance.md +++ b//eks/latest/best-practices/aiml-performance.md @@ -5 +5 @@ -Application Scaling and PerformanceOptimize Image Pull Performance +Application Scaling and PerformancePre-caching Container ImagesPulling Images with SOCI snapshotterReducing Container Image SizesHandling ML Model Artifacts in Deployments @@ -102 +102 @@ Deploying and serving machine learning (ML) models on Amazon EKS requires select -#### [Pre-caching Container Images](https://docs.aws.amazon.com/eks/latest/best-practices/aiml-performance.html#_reduce_container_startup_times_by_preloading_container_images_into_data_volumes) +## Pre-caching Container Images @@ -104 +104 @@ Deploying and serving machine learning (ML) models on Amazon EKS requires select -Large container images (e.g., models like PyTorch) can cause cold start delays that impact latency. For latency-sensitive workloads, like real-time inference workloads scaled horizontally and quick pod startup is critical, we recommend preloading container images to minimize initialization delays. Consider the following approaches from least to most recommended: +Large container images (such as those containing models like PyTorch) can cause cold start delays that impact latency. For latency-sensitive workloads, like real-time inference workloads scaled horizontally and quick pod startup is critical, we recommend preloading container images to minimize initialization delays. Consider the following approaches from least to most recommended: @@ -106 +106 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t -**Using the Container Runtime Cache to Pre-pull Images** +### Using the Container Runtime Cache to Pre-pull Images @@ -108 +108 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t - * You can pre-pull container images onto nodes using Kubernetes resources (e.g., DaemonSet or Deployment) to populate the node’s container runtime cache. The container runtime cache is the local storage managed by the container runtime (e.g., [containerd](https://containerd.io/)) where images are stored after being pulled from a registry. Pre-pulling ensures images are available locally, avoiding download delays during pod startup. This approach is useful when EBS snapshots are not preconfigured or when image pre-pulling is preferred. See the [AWS Samples GitHub repository](https://github.com/aws-samples/aws-do-eks/tree/main/Container-Root/eks/deployment/prepull) for examples of pre-pulling images. Note that alternatives like lazy loading with the [SOCI Snapshotter](https://github.com/awslabs/soci-snapshotter) (a containerd plugin for partial image pulls) can complement these methods, though it requires custom setup and may not suit all scenarios. Using the container runtime cache comes with the following pros and cons: +You can pre-pull container images onto nodes using Kubernetes resources (e.g., DaemonSet or Deployment) to populate the node’s container runtime cache. The container runtime cache is the local storage managed by the container runtime (e.g., [containerd](https://containerd.io/) where images are stored after being pulled from a registry. Pre-pulling ensures images are available locally, avoiding download delays during pod startup. This approach is particularly useful when images change often (e.g., frequent updates), when EBS snapshots are not preconfigured, when building an EBS volume would be more time-consuming than direct pulling from a container registry, or when nodes are already in the cluster and need to spin up pods on-demand using one of several possible images. @@ -110,28 +110 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t - * **Pros** : - - * No need to manage EBS snapshots. - - * With DaemonSets you always get the latest container image version. - - * More flexible, as images can be updated without recreating snapshots. - - * Still reduces pod startup time by ensuring images are already on the node. - - * **Cons** : - - * Initial pulling of large images can still take time, though it’s done in advance. - - * May not be as efficient as EBS snapshots for very large images (over 10 GB), since pulling is still required, albeit not at pod startup. - - * With DaemonSets, an image is pre-pulled to all nodes where the pod might run. For example, if 1000 nodes were only running one instance of a pod, space is consumed on all 1000 nodes just to run the one instance on one node. - - * For realtime inference workloads where nodes scale in/out, new nodes added by tools like Cluster Autoscaler may schedule workload pods **before** the pre-pull DaemonSet completes image pulling. This can cause the initial pod on the new node to trigger the pull anyway, potentially delaying startup and impacting low-latency requirements. - - * Kubelet image garbage collection can affect pre-pulled images by removing unused ones when disk usage exceeds certain thresholds or if they exceed a configured maximum unused age. In scale-in/out patterns, this may evict images on idle nodes, which requires re-pulls during subsequent scale-ups and reducing the reliability of the cache for bursty workloads. - - - - -**Using EBS Snapshots** - - * You can take an Amazon Elastic Block Store (EBS) snapshot of cached container images and reuse this snapshot for EKS worker nodes. This ensures images are prefetched locally upon node startup, reducing pod initialization time. See this [Reduce container startup time on Amazon EKS with Bottlerocket data volume](https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/) for more information using Karpenter and this [EKS Terraform Blueprints](https://aws-ia.github.io/terraform-aws-eks-blueprints/patterns/machine-learning/ml-container-cache/) for managed node groups. We recommend automating the creation of EBS snapshots as part of your CI/CD pipeline to keep them up-to-date with the latest container image versions. Using EBS snapshots comes with the following pros and cons: +Pre-pulling all variants ensures fast startup time regardless of which image is needed. For example, in a massively parallel ML workload requiring 100,000 small models built using 10 different techniques, pre-pulling 10 images via DaemonSet across a large cluster (e.g., thousands of nodes) minimizes pod startup time, enabling completion in under 10 seconds by avoiding on-demand pulls. Using the container runtime cache approach eliminates the need to manage EBS snapshots, ensures you always get the latest container image version with DaemonSets, but for real-time inference workloads where nodes scale in/out, new nodes added by tools like Cluster Autoscaler may schedule workload pods before the pre-pull DaemonSet completes image pulling. This can cause the initial pod on the new node to trigger the pull anyway, potentially delaying startup and impacting low-latency requirements. Additionally, kubelet image garbage collection can affect pre-pulled images by removing unused ones when disk usage exceeds certain thresholds or if they exceed a configured maximum unused age. In scale-in/out patterns, this may evict images on idle nodes, which requires re-pulls during subsequent scale-ups and reducing the reliability of the cache for bursty workloads. @@ -139,15 +112 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t - * **Pros** : - - * Eliminates the need to pull large container images at pod startup, significantly reducing startup time (e.g., from 10-15 minutes to seconds for images larger than 10 GB). - - * Ensures images are available locally upon node startup. - - * Particularly beneficial for inference workloads with large container images. - - * **Cons** : - - * Requires maintaining and updating EBS snapshots with every image or version upgrade. - - * Requires extra steps to make sure all your workloads use the latest container image version. - - * Involves additional operational activities to create and manage snapshots. +See [AWS GitHub repository](https://github.com/aws-samples/aws-do-eks/tree/main/Container-Root/eks/deployment/prepull) for examples of pre-pulling images into the container runtime cache. @@ -155 +114 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t - * May include unnecessary images if not properly managed, though this can be mitigated with proper node pool configuration. +### Using EBS Snapshots to Pre-pull Images @@ -156,0 +116 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t +You can take an Amazon Elastic Block Store (EBS) snapshot of cached container images and reuse this snapshot for EKS worker nodes. This ensures images are prefetched locally upon node startup, reducing pod initialization time. See [Reduce container startup time on Amazon EKS with Bottlerocket data volume](https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/) for more information using Karpenter and [EKS Terraform Blueprints for managed node groups](https://aws-ia.github.io/terraform-aws-eks-blueprints/patterns/machine-learning/ml-container-cache/). @@ -157,0 +118 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t +We recommend automating the creation of EBS snapshots as part of your CI/CD pipeline to keep them up-to-date with the latest container image versions. Note that alternatives like lazy loading with the [SOCI Snapshotter](https://github.com/awslabs/soci-snapshotter) can complement these methods, though it requires custom setup. Additionally, you can preload container images into Bottlerocket volumes. Using the EBS snapshot approach eliminates the need to pull large container images at pod startup, significantly reducing startup time (e.g., from 10-15 minutes to seconds for images larger than 10 GB), but it requires additional maintenance, as you need to update the EBS snapshot with every image or version upgrade. @@ -158,0 +120 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t +To learn more, see [Using containerd snapshotter](https://awslabs.github.io/ai-on-eks/docs/guidance/container-startup-time/accelerate-pull-process/containerd-snapshotter) and [Preload container images into Bottlerocket data volumes with EBS Snapshots](https://awslabs.github.io/ai-on-eks/docs/guidance/container-startup-time/accelerate-pull-process/prefecthing-images-on-br) in the AI on EKS Workshop. @@ -160 +122 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t -## Optimize Image Pull Performance +## Pulling Images with SOCI snapshotter @@ -162,9 +124 @@ Large container images (e.g., models like PyTorch) can cause cold start delays t -We strongly recommend optimizing container image pull performance for Amazon EKS clusters running AI/ML workloads. Using large, unoptimized base images or inefficient layer ordering can lead to slow pod startup times, increased resource consumption, and degraded inference latency. - -### Optimize Image Builds - -To address this, adopt small, lightweight base images with minimal dependencies, tailored to your workloads. You can also consider the AWS Deep Learning Containers (DLCs) which are pre-built container images that make it easier to run popular deep learning frameworks (e.g., [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/)). To learn more about building a custom image, see [Customize Deep Learning Containers](https://docs.aws.amazon.com/deep-learning-containers/latest/devguide/deep-learning-containers-custom-images.html). When building custom images, consider lightweight base images and add only necessary libraries to keep images lean. Use multi-stage builds to reduce layer size and optimize layer ordering for efficient caching. For more details, see the [Docker Best Practices for Building Images](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/). - -### Optimize Node Configuration and Runtime - -For very large images that you can’t easily minimize, we recommend using the open source Seekable OCI (SOCI) snapshotter configured in parallel pull and unpack mode. This solution lets you use existing images without rebuilding or modifying your build pipelines. This option is especially effective when deploying workloads with very large images to high performance EC2 compute instances. It works well with high-throughput networking and high performance storage configurations as is typical with scaled AI/ML workloads. +For very large images that you can’t easily minimize, you can use the open source Seekable OCI (SOCI) snapshotter configured in parallel pull and unpack mode. This solution lets you use existing images without rebuilding or modifying your build pipelines. This option is especially effective when deploying workloads with very large images to high performance EC2 compute instances. It works well with high-throughput networking and high performance storage configurations as is typical with scaled AI/ML workloads. @@ -186 +140 @@ SOCI parallel pull mode uses a dual-threshold control system with configurable p -#### References +**References** @@ -197 +151,9 @@ SOCI parallel pull mode uses a dual-threshold control system with configurable p -### Reduce Container Startup Times by Preloading Container Images into Data Volumes +## Reducing Container Image Sizes + +Reducing the size of container images during startup is another way to make images smaller. You can make reductions at every step of the container image build process. To start, choose base images that contain the least number of dependencies required. During image builds, include only the essential libraries and artifacts that are required. When building images, try combining multiple RUN or COPY commands to create a smaller number of larger layers. + +For AI/ML frameworks, use multi-stage builds to separate build and runtime, copying only required artifacts (e.g., via COPY —from= for registries or local contexts), and select variants like runtime-only images (e.g., `pytorch/pytorch:2.7.1-cuda11.8-cudnn9-runtime` at 3.03 GB vs. devel at 6.66 GB). To learn more, see [Reducing container image size](https://awslabs.github.io/ai-on-eks/docs/guidance/container-startup-time/reduce-container-image-size) in the AI on EKS Workshop. + +## Handling ML Model Artifacts in Deployments + +A key decision is how to handle the ML model artifacts (e.g., weights, configurations) themselves. The choice impacts image size, deployment speed, model update frequency, and operational overhead. Not that when referring to storing the "model", we are referring to the model artifacts (e.g., trained parameters, model weights). There are three approaches to handling ML model artifacts on Amazon EKS. Each has its trade-offs, and the best one depends on your model’s size, update cadence, and infrastructure needs. Consider the following approaches from least to most recommended: @@ -199 +161 @@ SOCI parallel pull mode uses a dual-threshold control system with configurable p -For machine learning workloads requiring low pod startup latency, such as real-time inference, we recommend preloading container images to minimize initialization delays. Large container images can slow pod startup, especially on nodes with limited bandwidth. In addition to using minimal base images, multi-stage builds, and lazy-loading techniques, consider the following approaches to preload images in Amazon EKS. In addition to using minimal base images, multi-stage builds, and lazy-loading techniques, consider the following options: + * **Baking the model into the container image** : Copy the model files (e.g., .safetensors, .pth, .h5) into the container image (e.g., Dockerfile) during image build. The model is part of the immutable image. We recommend using this approach for smaller models with infrequent updates. This ensures consistency and reproducibility, no loading delay, simplifies dependency management, but results in larger image sizes, slowing builds and pushes, requires rebuilding and redeploying for model updates, and it is not ideal for large models due to registry pull throughput. @@ -201 +163 @@ For machine learning workloads requiring low pod startup latency, such as real-t - * **Pre-load images using EBS snapshots** : Take an Amazon Elastic Block Store (EBS) snapshot of cached container images and reuse this snapshot for EKS worker nodes. Though this adds additional operational activities it ensures images are prefetched locally upon node startup, reducing pod initialization time. See this [Reduce container startup time on Amazon EKS with Bottlerocket data volume](https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/) for more information using Karpenter and this [EKS Terraform Blueprints](https://aws-ia.github.io/terraform-aws-eks-blueprints/patterns/machine-learning/ml-container-cache/) for managed node groups. + * **Downloading the model at runtime** : At container startup, the application downloads the model from external storage (e.g., Amazon S3, backed by S3 CRT for optimized high-throughput transfers using methods such as Mountpoint for S3 CSI driver, AWS S3 CLI, or s5cmd OSS CLI) via scripts in an init container or entrypoint. We recommend starting with this approach for large models with frequent updates. This keeps container images focused on code/runtime, enables easy model updates without rebuilds, supports versioning via storage metadata, but it introduces potential network failures (requires retry logic), it requires authentication and caching. @@ -203 +164,0 @@ For machine learning workloads requiring low pod startup latency, such as real-t - * **Pre-pull images into container runtime cache** : Pre-pull container images onto nodes using Kubernetes resources (e.g., DaemonSet or Deployment) to populate the node’s container runtime cache. The container runtime cache is the local storage managed by the container runtime (e.g., [containerd](https://containerd.io/)) where images are stored after being pulled from a registry. Pre-pulling ensures images are available locally, avoiding download delays during pod startup. This approach is useful when EBS snapshots are not preconfigured or when image pre-pulling is preferred. Test this approach in a staging environment to validate latency improvements. See the [AWS Samples GitHub repository](https://github.com/aws-samples/aws-do-eks/tree/main/Container-Root/eks/deployment/prepull) for examples of pre-pulling images. @@ -206,0 +168 @@ For machine learning workloads requiring low pod startup latency, such as real-t +To learn more, see [Accelerating pull process](https://awslabs.github.io/ai-on-eks/docs/guidance/container-startup-time/accelerate-pull-process) in the AI on EKS Workshop.