AWS fsx documentation change
Summary
Restructured performance documentation, removed detailed performance tables and 'Aggregate file system performance' section, added new sections about storage class characteristics and intelligent tiering
Security assessment
Changes focus on reorganizing performance documentation and adding new storage class information without any mention of security vulnerabilities, access controls, or encryption. The modifications are purely related to performance characteristics and architectural explanations.
Diff
diff --git a/fsx/latest/LustreGuide/performance.md b/fsx/latest/LustreGuide/performance.md index 60ae4fd6d..aa6fb92cd 100644 --- a//fsx/latest/LustreGuide/performance.md +++ b//fsx/latest/LustreGuide/performance.md @@ -5 +5 @@ -How FSx for Lustre file systems workAggregate file system performanceFile system metadata performanceThroughput to individual client instancesFile system storage layoutStriping data in your file systemMonitoring performance and usagePerformance tips +OverviewHow FSx for Lustre file systems workFile system metadata performanceThroughput to individual client instancesFile system storage layoutStriping data in your file systemMonitoring performance and usage @@ -9 +9 @@ How FSx for Lustre file systems workAggregate file system performanceFile system -Amazon FSx for Lustre, built on Lustre, the popular high-performance file system, provides scale-out performance that increases linearly with a file system’s size. Lustre file systems scale horizontally across multiple file servers and disks. This scaling gives each client direct access to the data stored on each disk to remove many of the bottlenecks present in traditional file systems. Amazon FSx for Lustre builds on the Lustre scalable architecture to support high levels of performance across large numbers of clients. +This chapter provides Amazon FSx for Lustre performance topics, including some important tips and recommendations for maximizing the performance of your file system. @@ -13 +13 @@ Amazon FSx for Lustre, built on Lustre, the popular high-performance file system - * How FSx for Lustre file systems work + * Overview @@ -15 +15 @@ Amazon FSx for Lustre, built on Lustre, the popular high-performance file system - * Aggregate file system performance + * How FSx for Lustre file systems work @@ -27,16 +27 @@ Amazon FSx for Lustre, built on Lustre, the popular high-performance file system - * Performance tips - - - - -## How FSx for Lustre file systems work - -Each FSx for Lustre file system consists of the file servers that the clients communicate with, and a set of disks attached to each file server that store your data. Each file server employs a fast, in-memory cache to enhance performance for the most frequently accessed data. HDD-based file systems can also be provisioned with an SSD-based read cache to further enhance performance for the most frequently accessed data. When a client accesses data that's stored in the in-memory or SSD cache, the file server doesn't need to read it from disk, which reduces latency and increases the total amount of throughput you can drive. The following diagram illustrates the paths of a write operation, a read operation served from disk, and a read operation served from in-memory or SSD cache. - - - -When you read data that is stored on the file server's in-memory or SSD cache, file system performance is determined by the network throughput. When you write data to your file system, or when you read data that isn't stored on the in-memory cache, file system performance is determined by the lower of the network throughput and disk throughput. - -When you provision an HDD Lustre file system with an SSD cache, Amazon FSx creates an SSD cache that is automatically sized to 20 percent of the file system's HDD storage capacity. Doing this provides sub-millisecond latencies and higher IOPS for frequently accessed files. - -## Aggregate file system performance + * [Performance characteristics of SSD and HDD storage classes](./ssd-storage.html) @@ -44 +29 @@ When you provision an HDD Lustre file system with an SSD cache, Amazon FSx creat -The throughput that an FSx for Lustre file system supports is proportional to its storage capacity. Amazon FSx for Lustre file systems scale to hundreds of GBps of throughput and millions of IOPS. Amazon FSx for Lustre also supports concurrent access to the same file or directory from thousands of compute instances. This access enables rapid data checkpointing from application memory to storage, which is a common technique in high performance computing (HPC). You can increase the amount of storage and throughput capacity as needed at any time after you create the file system. For more information, see [Managing storage capacity](./managing-storage-capacity.html). + * [Performance characteristics of Intelligent-Tiering storage class](./intelligent-tiering-file-systems.html) @@ -46 +31 @@ The throughput that an FSx for Lustre file system supports is proportional to it -FSx for Lustre file systems provide burst read throughput using a network I/O credit mechanism to allocate network bandwidth based on average bandwidth utilization. The file systems accrue credits when their network bandwidth usage is below their baseline limits, and can use these credits when they perform network data transfers. + * [Performance tips](./performance-tips.html) @@ -48 +32,0 @@ FSx for Lustre file systems provide burst read throughput using a network I/O cr -The following tables show performance that the FSx for Lustre deployment options are designed for. @@ -50,8 +33,0 @@ The following tables show performance that the FSx for Lustre deployment options -File system performance for SSD storage options Deployment Type | **Network throughput (MBps/TiB of storage provisioned)** | **Network IOPS (IOPS/TiB of storage provisioned)** | **Cache storage (GiB of RAM/TiB of storage provisioned)** | **Disk latencies per file operation (milliseconds, P50)** | **Disk throughput (MBps/TiB of storage or SSD cache provisioned)** ----|---|---|---|---|--- -****| **Baseline** | **Burst** | ****| ****| ****| **Baseline** | **Burst** -SCRATCH_2 | 200 | 1300 | Tens of thousands baselineHundreds of thousands burst | 6.7 | Metadata: sub-ms Data: sub-ms | 200 (read) 100 (write) | ‐ -PERSISTENT-125 | 320 | 1300 | 3.4 | 125 | 500 -PERSISTENT-250 | 640 | 1300 | 6.8 | 250 | 500 -PERSISTENT-500 | 1300 | ‐ | 13.7 | 500 | ‐ -PERSISTENT-1000 | 2600 | ‐ | 27.3 | 1000 | ‐ @@ -59,9 +34,0 @@ PERSISTENT-1000 | 2600 | ‐ | 27.3 | 1000 | ‐ -File system performance for HDD storage options Deployment Type | **Network throughput (MBps/TiB of storage or SSD cache provisioned)** | **Network IOPS (IOPS/TiB of storage provisioned)** | **Cache storage (GiB of RAM/TiB of storage provisioned)** | **Disk latencies per file operation (milliseconds, P50)** | **Disk throughput (MBps/TiB of storage or SSD cache provisioned)** ----|---|---|---|---|--- -****| **Baseline** | **Burst** | | **Baseline** | **Burst** -PERSISTENT-12 -HDD storage | 40 | 375* | Tens of thousands baseline Hundreds of thousands burst | 0.4 memory | Metadata: sub-ms Data: single-digit ms | 12 | 80 (read) 50 (write) -SSD read cache | 200 | 1,900 | 200 SSD cache | Data: sub-ms | 200 | - -PERSISTENT-40 -HDD storage | 150 | 1,300* | Tens of thousands baseline Hundreds of thousands burst | 1.5 | Metadata: sub-ms Data: single-digit ms | 40 | 250 (read) 150 (write) -SSD read cache | 750 | 6500 | 200 SSD cache | Data: sub-ms | 200 | - @@ -69,6 +36 @@ SSD read cache | 750 | 6500 | 200 SSD cache | Data: sub-ms | 200 | - -File system performance for previous generation SSD storage options Deployment Type | **Network throughput (MBps per TiB of storage provisioned)** | **Network IOPS (IOPS per TiB of storage provisioned)** | **Cache storage (GiB per TiB of storage provisioned)** | **Disk latencies per file operation (milliseconds, P50)** | **Disk throughput (MBps per TiB of storage or SSD cache provisioned)** ----|---|---|---|---|--- -****| **Baseline** | **Burst** | ****| ****| ****| **Baseline** | **Burst** -PERSISTENT-50 | 250 | 1,300* | Tens of thousands baselineHundreds of thousands burst | 2.2 RAM | Metadata: sub-ms Data: sub-ms | 50 | 240 -PERSISTENT-100 | 500 | 1,300* | 4.4 RAM | 100 | 240 -PERSISTENT-200 | 750 | 1,300* | 8.8 RAM | 200 | 240 +## Overview @@ -76 +38 @@ PERSISTENT-200 | 750 | 1,300* | 8.8 RAM | 200 | 240 -###### Note +Amazon FSx for Lustre, built on Lustre, the popular high-performance file system, provides scale-out performance that increases linearly with a file system’s size. Lustre file systems scale horizontally across multiple file servers and disks. This scaling gives each client direct access to the data stored on each disk to remove many of the bottlenecks present in traditional file systems. Amazon FSx for Lustre builds on the Lustre scalable architecture to support high levels of performance across large numbers of clients. @@ -78 +40 @@ PERSISTENT-200 | 750 | 1,300* | 8.8 RAM | 200 | 240 -* Persistent file systems in the following AWS Regions provide network burst up to 530 MBps per TiB of storage: Africa (Cape Town), Asia Pacific (Hong Kong), Asia Pacific (Osaka), Asia Pacific (Singapore), Canada (Central), Europe (Frankfurt), Europe (London), Europe (Milan), Europe (Stockholm), Middle East (Bahrain), South America (São Paulo), China, and US West (Los Angeles). +## How FSx for Lustre file systems work @@ -80 +42 @@ PERSISTENT-200 | 750 | 1,300* | 8.8 RAM | 200 | 240 -### Example: Aggregate baseline and burst throughput +Each FSx for Lustre file system consists of the file servers that the clients communicate with, and a set of disks attached to each file server that store your data. Each file server employs a fast, in-memory cache to enhance performance for the most frequently accessed data. Depending on the storage class, your file server can be provisioned with an optional SSD read cache. When a client accesses data that's stored in the in-memory or SSD cache, the file server doesn't need to read it from disk, which reduces latency and increases the total amount of throughput you can drive. The following diagram illustrates the paths of a write operation, a read operation served from disk, and a read operation served from in-memory or SSD cache. @@ -82 +44 @@ PERSISTENT-200 | 750 | 1,300* | 8.8 RAM | 200 | 240 -The following example illustrates how storage capacity and disk throughput impact file system performance. + @@ -84 +46 @@ The following example illustrates how storage capacity and disk throughput impac -A persistent file system with a storage capacity of 4.8 TiB and 50 MBps per TiB of throughput per unit of storage provides an aggregate baseline disk throughput of 240 MBps and a burst disk throughput of 1.152 GBps. +When you read data that is stored on the file server's in-memory or SSD cache, file system performance is determined by the network throughput. When you write data to your file system, or when you read data that isn't stored on the in-memory cache, file system performance is determined by the lower of the network throughput and disk throughput. @@ -86 +48 @@ A persistent file system with a storage capacity of 4.8 TiB and 50 MBps per TiB -Regardless of file system size, Amazon FSx for Lustre provides consistent, sub-millisecond latencies for file operations. +To learn more about network throughput, disk throughput, and IOPS characteristics of SSD and HDD storage classes, see [Performance characteristics of SSD and HDD storage classes](./ssd-storage.html) and [Performance characteristics of Intelligent-Tiering storage class](./intelligent-tiering-file-systems.html). @@ -90 +52 @@ Regardless of file system size, Amazon FSx for Lustre provides consistent, sub-m -File system metadata IO operations per second (IOPS) determine the number of files and directories that you can create, list, read, and delete per second. Metadata IOPS are automatically provisioned on FSx for Lustre file systems based on the storage capacity that you provision. +File system metadata IO operations per second (IOPS) determine the number of files and directories that you can create, list, read, and delete per second. @@ -92 +54 @@ File system metadata IO operations per second (IOPS) determine the number of fil -Persistent 2 file systems allow you to provision Metadata IOPS independent from storage capacity and provide increased visibility into the number and type of metadata IOPS client instances are driving on your file system. +Persistent 2 file systems allow you to provision Metadata IOPS independent from storage capacity and provide increased visibility into the number and type of metadata IOPS client instances are driving on your file system. With SSD file systems, Metadata IOPS are automatically provisioned based on the storage capacity that you provision. Automatic mode is not supported on Intelligent-Tiering file systems. @@ -94 +56 @@ Persistent 2 file systems allow you to provision Metadata IOPS independent from -With FSx for Lustre Persistent 2 file systems, the number of metadata IOPS you provision and the type of metadata operation determine the rate of metadata operations that your file system can support. The level of metadata IOPS you provision determines the number of IOPS provisioned for your file system's metadata disks. +With FSx for Lustre Persistent 2 file systems, the number of Metadata IOPS you provision and the type of metadata operation determine the rate of metadata operations that your file system can support. The level of metadata IOPS you provision determines the number of IOPS provisioned for your file system's metadata disks. @@ -103 +65 @@ Directory Delete | 0.2 -You can choose to provision metadata IOPS using Automatic mode or User-provisioned mode. In Automatic mode, Amazon FSx automatically provisions metadata IOPS based on the storage capacity of your file system according to the table below: +For SSD file systems, you can choose to provision metadata IOPS using Automatic mode. In Automatic mode, Amazon FSx automatically provisions metadata IOPS based on the storage capacity of your file system according to the table below: @@ -113 +75,10 @@ File system storage capacity | Included metadata IOPS in Automatic mode -In User-provisioned mode, you can optionally choose to specify the number of metadata IOPS to provision. You pay for Metadata IOPS provisioned above the default number of Metadata IOPS for your file system. +In User-provisioned mode, you can optionally choose to specify the number of metadata IOPS to provision. Valid values are as follows: + + * For SSD file systems, valid values are `1500`, `3000`, `6000`, `12000`, and multiples of `12000` up to a maximum of `192000`. + + * For Intelligent-Tiering file systems, valid values are `6000` and `12000`. + + + + +For information about how to configure Metadata IOPS, see [Managing metadata performance](./managing-metadata-performance.html). Note that you pay for Metadata IOPS provisioned above the default number of Metadata IOPS for your file system. @@ -131 +102 @@ EFA-enabled | EFA with GDS | 1200 Gbps -* The traffic between an individual client instance and an individual FSx for Lustre object storage server is limited to 5 Gbps. Refer to the [Prerequisites](./getting-started.html#prerequisites) for the number of object storage servers underpinning your FSx for Lustre file system. +* The traffic between an individual client instance and an individual FSx for Lustre object storage server is limited to 5 Gbps. Refer to the [IP addresses for file systems](./using-fsx-lustre.html#ip-addesses-for-fs) for the number of object storage servers underpinning your FSx for Lustre file system. @@ -135 +106 @@ EFA-enabled | EFA with GDS | 1200 Gbps -All file data in Lustre is stored on storage volumes called _object storage targets_ (OSTs). All file metadata (including file names, timestamps, permissions, and more) is stored on storage volumes called _metadata targets_ (MDTs). Amazon FSx for Lustre file systems are composed of one or more MDTs and multiple OSTs. Each OST is approximately 1 to 2 TiB in size, depending on the file system's deployment type. Amazon FSx for Lustre spreads your file data across the OSTs that make up your file system to balance storage capacity with throughput and IOPS load. +All file data in Lustre is stored on storage volumes called _object storage targets_ (OSTs). All file metadata (including file names, timestamps, permissions, and more) is stored on storage volumes called _metadata targets_ (MDTs). Amazon FSx for Lustre file systems are composed of one or more MDTs and multiple OSTs. Amazon FSx for Lustre spreads your file data across the OSTs that make up your file system to balance storage capacity with throughput and IOPS load. @@ -263,52 +233,0 @@ For more information on monitoring your file system’s performance, see [Monito -## Performance tips - -When using Amazon FSx for Lustre, keep the following performance tips in mind. For service limits, see [Quotas for Amazon FSx for Lustre](./limits.html). - - * **Average I/O size** – Because Amazon FSx for Lustre is a network file system, each file operation goes through a round trip between the client and Amazon FSx for Lustre, incurring a small latency overhead. Due to this per-operation latency, overall throughput generally increases as the average I/O size increases, because the overhead is amortized over a larger amount of data. - - * **Request model** – By enabling asynchronous writes to your file system, pending write operations are buffered on the Amazon EC2 instance before they are written to Amazon FSx for Lustre asynchronously. Asynchronous writes typically have lower latencies. When performing asynchronous writes, the kernel uses additional memory for caching. A file system that has enabled synchronous writes issues synchronous requests to Amazon FSx for Lustre. Every operation goes through a round trip between the client and Amazon FSx for Lustre. - -###### Note - -Your chosen request model has tradeoffs in consistency (if you're using multiple Amazon EC2 instances) and speed. - - * **Limit directory size** – To achieve optimal metadata performance on Persistent 2 FSx for Lustre file systems, limit each directory to less than 100K files. Limiting the number of files in a directory reduces the time required for the file system to acquire a lock on the parent directory. - - * **Amazon EC2 instances** – Applications that perform a large number of read and write operations likely need more memory or computing capacity than applications that don't. When launching your Amazon EC2 instances for your compute-intensive workload, choose instance types that have the amount of these resources that your application needs. The performance characteristics of Amazon FSx for Lustre file systems don't depend on the use of Amazon EBS–optimized instances. - - * **Recommended client instance tuning for optimal performance** - - 1. For client instance types with memory of more than 64 GiB, we recommend applying the following tuning: - - sudo lctl set_param ldlm.namespaces.*.lru_max_age=600000 - sudo lctl set_param ldlm.namespaces.*.lru_size=<100 * number_of_CPUs> - - 2. For client instance types with more than 64 vCPU cores, we recommend applying the following tuning: - - echo "options ptlrpc ptlrpcd_per_cpt_max=32" >> /etc/modprobe.d/modprobe.conf - echo "options ksocklnd credits=2560" >> /etc/modprobe.d/modprobe.conf - - # reload all kernel modules to apply the above two settings - sudo reboot - -After the client is mounted, the following tuning needs to be applied: - - sudo lctl set_param osc.*OST*.max_rpcs_in_flight=32 - sudo lctl set_param mdc.*.max_rpcs_in_flight=64 - sudo lctl set_param mdc.*.max_mod_rpcs_in_flight=50 - -Note that `lctl set_param` is known to not persist over reboot. Since these parameters cannot be set permanently from the client side, it is recommended to implement a boot cron job to set the configuration with the recommended tunings. - - * **Workload balance across OSTs** – In some cases, your workload isn’t driving the aggregate throughput that your file system can provide (200 MBps per TiB of storage). If so, you can use CloudWatch metrics to troubleshoot if performance is affected by an imbalance in your workload’s I/O patterns. To identify if this is the cause, look at the Maximum CloudWatch metric for Amazon FSx for Lustre. - -In some cases, this statistic shows a load at or above 240 MBps of throughput (the throughput capacity of a single 1.2-TiB Amazon FSx for Lustre disk). In such cases, your workload is not evenly spread out across your disks. If this is the case, you can use the `lfs setstripe` command to modify the striping of files your workload is most frequently accessing. For optimal performance, stripe files with high throughput requirements across all the OSTs comprising your file system. - -If your files are imported from a data repository, you can take another approach to stripe your high-throughput files evenly across your OSTs. To do this, you can modify the `ImportedFileChunkSize` parameter when creating your next Amazon FSx for Lustre file system. - -For example, suppose that your workload uses a 7.0-TiB file system (which is made up of 6x 1.17-TiB OSTs) and needs to drive high throughput across 2.4-GiB files. In this case, you can set the `ImportedFileChunkSize` value to `(2.4 GiB / 6 OSTs) = 400 MiB` so that your files are spread evenly across your file system’s OSTs. - - * **Lustre client for Metadata IOPS** – If your file system has a metadata configuration specified, we recommend you install a Lustre 2.15 client or a Lustre 2.12 client with one of these OS versions: Amazon Linux 2023; Amazon Linux 2; Red Hat/Rocky Linux 8.9, 8.10, or 9.x; CentOS 8.9 or 8.10; Ubuntu 22+ with 6.2, 6.5, or 6.8 kernel; or Ubuntu 20. - - - - @@ -323 +242 @@ Working with older deployment types -Accessing file systems +SSD and HDD storage classes