AWS eks documentation change
Summary
Added Amazon FSx for OpenZFS as a recommended storage option, expanded performance monitoring guidance using CloudWatch, and provided detailed configuration examples for FSx for OpenZFS integration.
Security assessment
The changes focus on performance optimization, storage scalability, and new service integration (FSx for OpenZFS). While monitoring and HA deployment types improve reliability, there is no explicit mention of security vulnerabilities, access controls, encryption, or incident response. The CIDR-based NFS export configuration in examples is a standard practice, not a security fix.
Diff
diff --git a/eks/latest/best-practices/aiml-storage.md b/eks/latest/best-practices/aiml-storage.md index b56d9fc12..17a6bf1e0 100644 --- a//eks/latest/best-practices/aiml-storage.md +++ b//eks/latest/best-practices/aiml-storage.md @@ -13 +13 @@ Data Management and Storage -AI/ML workloads often require access to large model artifacts (e.g., trained weights, configurations), and pods need a reliable, scalable way to access these without embedding them in container images, which can increase image sizes and Container registry pull times. To reduce operational overhead of managing volume mounts we recommend deploying AI models to pods by mounting Amazon storage services (e.g., S3, FSx for Lustre, EFS) as Persistent Volumes (PVs) using their respective CSI drivers. For implementation details, see subsequent topics in this section. +AI/ML workloads often require access to large model artifacts (e.g., trained weights, configurations), and pods need a reliable, scalable way to access these without embedding them in container images, which can increase image sizes and Container registry pull times. To reduce operational overhead of managing volume mounts we recommend deploying AI models to pods by mounting Amazon storage services (e.g., S3, FSx for Lustre, FSx for OpenZFS, EFS) as Persistent Volumes (PVs) using their respective CSI drivers. For implementation details, see subsequent topics in this section. @@ -35,0 +36,2 @@ The choice of AWS Storage service depends on your deployment architecture, scale +**Monitoring performance** Poor disk performance can delay container image reads, increase pod startup latency, and degrade inference or training throughput. Use [Amazon CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html) to monitor performance metrics for your AWS storage services. When you identify performance bottlenecks, modify your storage configuration parameters to optimize performance. + @@ -49 +51 @@ You can [Install the FSx for Lustre CSI driver](https://docs.aws.amazon.com/eks/ -Performance considerations: +**Performance considerations:** @@ -135 +137 @@ Configure a pod to use an Persistent Volume Claim of `fsx-claim`: -For complete examples, see the [FSx for Lustre Driver Examples in GitHub](https://github.com/kubernetes-sigs/aws-fsx-csi-driver/tree/master/examples/kubernetes). +For complete examples, see the [FSx for Lustre Driver Examples in GitHub](https://github.com/kubernetes-sigs/aws-fsx-csi-driver/tree/master/examples/kubernetes). Monitor [Amazon FSx for Lustre performance metrics](https://docs.aws.amazon.com/fsx/latest/LustreGuide/monitoring-cloudwatch.html) using Amazon CloudWatch. When performance bottlenecks are identified, adjust your configuration parameters as needed. @@ -183,0 +186,101 @@ For more details, see the [Mountpoint for Amazon S3 CSI Driver](https://docs.aws +### Amazon FSx for OpenZFS persistent shared storage + +For scenarios involving multiple EC2 GPU compute instances with latency-sensitive workloads requiring high availability, high performance, cost sensitivity, and multiple pod deployments for different applications, we recommend Amazon FSx for OpenZFS. Some workload examples include real-time inference, reinforcement learning, and training generative adversarial networks. FSx for OpenZFS is particularly beneficial for workloads needing high performance access to a focused directory structure with small files using small IO data access patterns. Also, FSx for OpenZFS provides the flexibility to scale performance independently from storage capacity, helping you achieve optimal cost efficiency by matching storage size to actual needs while maintaining required performance levels + +The native [FSx for OpenZFS CSI driver](https://github.com/kubernetes-sigs/aws-fsx-openzfs-csi-driver/tree/main) allows for the creation of multiple PVCs to a single file system by creating multiple volumes. This reduces management overhead and maximizes the utilization of the file system’s throughput and IOPS through consolidated application pod deployments on a single file system. Additionally, it includes enterprise features like zero-copy snapshots, zero-copy clones, and user and group quotas which can be dynamically provisioned through the CSI driver. + +FSx for OpenZFS supports three different [deployment types](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/availability-durability.html#choosing-single-or-multi) upon creation: + + * **Single-AZ:** Lowest cost option with sub-millisecond latencies, but provides no high-availability at the file system or Availability Zone level. Recommended for development and test workloads or those which have high-availability at the application layer. + + * **Single-AZ (HA):** Provides high-availability at the file system level with sub-millisecond latencies. Recommended for highest performance workloads which require high-availability. + + * **Multi-AZ:** Provides high-availability at the file system level as well as across Availability Zones. Recommended for high-performance workloads that require the additional availability across Availability Zones. + + + + +Performance considerations: + + * **Deployment type:** If the additional availability across Availability Zones isn’t a requirement, consider using the Single-AZ (HA) deployment type. This deployment type provides up to 100% of the throughput for writes, maintains sub-millisecond latencies, and the Gen2 file systems have an additional NVMe cache for storing up to terrabytes of frequently accessed data. The Multi-AZ file systems provide up to 75% of the throughput for writes at an increased latency to accomodate for cross-AZ traffic. + + * **Throughput and IOPS:** Both the [throughput](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/managing-throughput-capacity.html) and [IOPS](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/managing-storage-capacity.html) configured for the file system can be scaled up or down post deployment. You can provision up to 10GB/s of disk throughput providing up to 21GB/s of cached data access. The IOPS can be scaled up to 400,000 from disk and the cache can provide over 1 million IOPS. Note that throughput scaling of a Single-AZ file system does cause a brief outage of the file system as no high-availability exists. Throughput scaling of a Single-AZ (HA) or Multi-AZ file system can be done non-disruptively. The SSD IOPS can be scaled once every six hours. + + * **Storage Class:** FSx for OpenZFS supports both the [SSD storage](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/performance-ssd.html) class as well as the [Intelligent-Tiering](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/performance-intelligent-tiering.html) storage class. For AI/ML workloads it is recommended to use the SSD storage class providing consistent performance to the workload keeping the CPU’s/GPU’s as busy as possible. + + * **Compression:** Enable the [LZ4 compression](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/performance.html#perf-data-compression) algorithm if you have a workload that can be compressed. This reduces the amount of data each file consumes in the cache allowing more data to be served directly from the cache as network throughput and IOPS reducing the load on the SSD disk. + + * **Record size:** Most AI/ML workloads will benefit from leaving the default 128KiB [record size](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/performance.html#record-size-performance). This value should only be reduced if the dataset consists of large files (above 10GiB) with consistent small block access below 128KiB from the application. + + + + +Once the file system is created, an associated root volume is automatically created by the service. It is best practice to store data within child volumes of the root volume on the file system. Using the [FSx for OpenZFS CSI driver](https://github.com/kubernetes-sigs/aws-fsx-openzfs-csi-driver/tree/main) you create an associated Persistent Volume Claim to dynamically create the child volume. + +Examples: + +A Storage Class (SC) definition for an FSx for OpenZFS volume, used to create a child volume of the root volume ($ROOT_VOL_ID) on an existing file system and export the volume to the VPC CIDR ($VPC_CIDR) using the NFS v4.2 protocol. + + + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: fsxz-vol-sc + provisioner: fsx.openzfs.csi.aws.com + parameters: + ResourceType: "volume" + ParentVolumeId: '"$ROOT_VOL_ID"' + CopyTagsToSnapshots: 'false' + DataCompressionType: '"LZ4"' + NfsExports: '[{"ClientConfigurations": [{"Clients": "$VPC_CIDR", "Options": ["rw","crossmnt","no_root_squash"]}]}]' + ReadOnly: 'false' + RecordSizeKiB: '128' + Tags: '[{"Key": "Name", "Value": "AI-ML"}]' + OptionsOnDeletion: '["DELETE_CHILD_VOLUMES_AND_SNAPSHOTS"]' + reclaimPolicy: Delete + allowVolumeExpansion: false + mountOptions: + - nfsvers=4.2 + - rsize=1048576 + - wsize=1048576 + - timeo=600 + - nconnect=16 + - async + +A dynamically created Persistent Volume Claim (PVC) against the fsxz-vol-sc created above. **Note** , the storage capacity allocated is 1Gi, this is required for FSx for OpenZFS volumes as noted in the [CSI driver FAQ](https://github.com/kubernetes-sigs/aws-fsx-openzfs-csi-driver/blob/main/docs/FAQ.md). The volume will be provided the full capacity provisioned to the file system with this configuration. If the volume capacity needs to be restricted you can do so using user or group quotas. + + + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: dynamic-vol-pvc + namespace: example + spec: + accessModes: + - ReadWriteMany + storageClassName: fsxz-vol-sc + resources: + requests: + storage: 1Gi + +Configure a pod to mount a volume using the Persistent Volume Claim (PVC) of dynamic-vol-pvc: + + + kind: Pod + apiVersion: v1 + metadata: + name: fsx-app + namespace: example + spec: + volumes: + - name: dynamic-vol-pv + persistentVolumeClaim: + claimName: dynamic-vol-pvc + containers: + - name: app + image: amazonlinux:2023 + command: ["/bin/sh"] + volumeMounts: + - mountPath: "/mnt/fsxz" + name: dynamic-vol-pv + @@ -210,12 +313 @@ To view performance specifications, see [Amazon EFS performance specifications]( -For complete examples of using Amazon EFS file system as a persistent Volume within your EKS cluster and Pods, refer to the [EFS CSI Driver Examples in GitHub](https://github.com/kubernetes-sigs/aws-efs-csi-driver/tree/master/examples/kubernetes). - -**Monitoring performance** Poor disk performance can delay container image reads, increase pod startup latency, and degrade inference or training throughput. We recommend the following methods to monitor the performance metrics of the respective AWS Storage service if bottlenecks occur and adjusting your configuration where required. - - * [Amazon FSx console and its performance metrics](https://docs.aws.amazon.com/fsx/latest/LustreGuide/monitoring-cloudwatch.html) to view the performance metrics related to your FSx file system. - - * [Access Amazon CloudWatch metrics for Amazon EFS](https://docs.aws.amazon.com/efs/latest/ug/accessingmetrics.html) to view the performance metrics related to your EFS file system. - - * [Monitoring Amazon S3 metrics with Amazon CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) to view performance details related to your S3 bucket. - - - +For complete examples of using Amazon EFS file system as a persistent Volume within your EKS cluster and Pods, refer to the [EFS CSI Driver Examples in GitHub](https://github.com/kubernetes-sigs/aws-efs-csi-driver/tree/master/examples/kubernetes). Monitor [Amazon EFS performance metrics](https://docs.aws.amazon.com/efs/latest/ug/accessingmetrics.html) using Amazon CloudWatch. When performance bottlenecks are identified, adjust your configuration parameters as needed.