AWS AmazonS3 documentation change
Summary
Restructured documentation to focus on S3 Express One Zone directory bucket mechanics, added detailed best practices for performance optimization including directory structure guidance, concurrency recommendations, and operational examples
Security assessment
Changes focus on performance optimization patterns (directory structure strategies, request parallelization, SDK usage) without mentioning security vulnerabilities, access controls, encryption, or security mechanisms. While sections like 'Use Gateway VPC endpoints' and session authentication touch on network architecture and authentication, they are presented in the context of performance optimization rather than security hardening.
Diff
diff --git a/AmazonS3/latest/userguide/s3-express-optimizing-performance-design-patterns.md b/AmazonS3/latest/userguide/s3-express-optimizing-performance-design-patterns.md index f18e1600f..a0ebc0f4c 100644 --- a//AmazonS3/latest/userguide/s3-express-optimizing-performance-design-patterns.md +++ b//AmazonS3/latest/userguide/s3-express-optimizing-performance-design-patterns.md @@ -5 +5 @@ -Co-locate S3 Express One Zone storage with your AWS compute resourcesDirectory bucketsDirectory bucket horizontal scaling request parallelizationPerformance troubleshooting +How directory buckets workBest practicesPerformance troubleshootingDirectory bucket operation and directory interaction examples @@ -7 +7 @@ Co-locate S3 Express One Zone storage with your AWS compute resourcesDirectory b -# Performance guidelines and design patterns for S3 Express One Zone +# Best practices to optimize S3 Express One Zone performance @@ -13 +13 @@ For performance guidelines for all other Amazon S3 storage classes and S3 genera -To obtain the best performance for your application when using the S3 Express One Zone storage class and directory buckets, we recommend the following guidelines and design patterns. +For optimal performance and scalability with S3 Express One Zone storage class and directory buckets in high-scale workloads, it's important to understand how directory buckets work differently from general purpose buckets. Then, we provide best practices to align your applications with the way directory buckets work. @@ -15 +15 @@ To obtain the best performance for your application when using the S3 Express On -###### Topics +## How directory buckets work @@ -17 +17 @@ To obtain the best performance for your application when using the S3 Express On - * Co-locate S3 Express One Zone storage with your AWS compute resources +Amazon S3 Express One Zone storage class can support workloads with up to 2,000,000 GET and up to 200,000 PUT transactions per second (TPS) per directory bucket. With S3 Express One Zone, data is stored in S3 directory buckets in Availability Zones. Objects in directory buckets are accessible within a hierarchical namespace, similar to a file system and in contrast to S3 general purpose buckets that have a flat namespace. Unlike general purpose buckets, directory buckets organize keys hierarchically into directories instead of prefixes. A prefix is a string of characters at the beginning of the object key name. You can use prefixes to organize your data and manage a flat object storage architecture in general purpose buckets. For more information, see [Organizing objects using prefixes](./using-prefixes.html). @@ -19 +19 @@ To obtain the best performance for your application when using the S3 Express On - * Directory buckets +In directory buckets, objects are organized in a hierarchical namespace using forward slash (`/`) as the only supported delimiter. When you upload an object with a key like `dir1/dir2/file1.txt`, the directories `dir1/` and `dir2/` are automatically created and managed by Amazon S3. Directories are created during `PutObject` or `CreateMultiPartUpload` operations and automatically removed when they become empty after `DeleteObject` or `AbortMultiPartUpload` operations. There is no upper limit to the number of objects and subdirectories in a directory. @@ -21 +21 @@ To obtain the best performance for your application when using the S3 Express On - * Directory bucket horizontal scaling request parallelization +The directories that are created when objects are uploaded to directory buckets can scale instantaneously to reduce the chance of HTTP `503 (Slow Down)` errors. This automatic scaling allows your applications to parallelize read and write requests within and across directories as needed. For S3 Express One Zone, individual directories are designed to support the maximum request rate of a directory bucket. There is no need to randomize key prefixes to achieve optimal performance as the system automatically distributes objects for even load distribution, but as a result, keys are not stored lexicographically in directory buckets. This is in contrast to S3 general purpose buckets where keys that are lexicographically closer are more likely to be co-located on the same server. @@ -23 +23 @@ To obtain the best performance for your application when using the S3 Express On - * Performance troubleshooting +For more information about examples of directory bucket operations and directory interactions, see Directory bucket operation and directory interaction examples. @@ -24,0 +25 @@ To obtain the best performance for your application when using the S3 Express On +## Best practices @@ -25,0 +27 @@ To obtain the best performance for your application when using the S3 Express On +Follow the best practices to optimize your directory bucket performance and help your workloads scale over time. @@ -26,0 +29 @@ To obtain the best performance for your application when using the S3 Express On +### Use directories that contain many entries (objects or subdirectories) @@ -28 +31 @@ To obtain the best performance for your application when using the S3 Express On -## Co-locate S3 Express One Zone storage with your AWS compute resources +Directory buckets deliver high performance by default for all workloads. For even greater performance optimization with certain operations, consolidating more entries (which are objects or subdirectories) into directories will lead to lower latency and higher request rate: @@ -30 +33 @@ To obtain the best performance for your application when using the S3 Express On -Each directory bucket is stored in a single Availability Zone that you select when you create the bucket. You can get started by creating a new directory bucket in an Availability Zone local to your compute workloads or resources. You can then immediately begin very low-latency reads and writes. Directory buckets are the first S3 buckets where you can choose the Availability Zone in an AWS Region to reduce latency between compute and storage. + * Mutating API operations, such as `PutObject`, `DeleteObject`, `CreateMultiPartUpload` and `AbortMultiPartUpload`, achieve optimal performance when implemented with fewer, denser directories containing thousands of entries, rather than with a large number of smaller directories. @@ -32 +35 @@ Each directory bucket is stored in a single Availability Zone that you select wh -If you access directory buckets across Availability Zones, latency will increase. To optimize performance, we recommend that you access a directory bucket from Amazon Elastic Container Service, Amazon Elastic Kubernetes Service, and Amazon Elastic Compute Cloud instances that are located in the same Availability Zone when possible. + * `ListObjectsV2` operations perform better when fewer directories need to be traversed to populate a page of results. @@ -34 +36,0 @@ If you access directory buckets across Availability Zones, latency will increase -## Directory buckets @@ -36 +37,0 @@ If you access directory buckets across Availability Zones, latency will increase -Each directory bucket can support up to 2 million transactions per second (TPS). Unlike general purpose buckets, directory buckets organize keys hierarchically into directories instead of prefixes. A prefix is a string of characters at the beginning of the object key name. You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories. @@ -38 +38,0 @@ Each directory bucket can support up to 2 million transactions per second (TPS). -Prefixes organize data in a flat namespace within general purpose buckets, and there are no limits to the number of prefixes within a general purpose bucket. Each prefix can achieve at least 3,500 `PUT`/`POST`/`DELETE` or 5,500 `GET`/`HEAD` requests per second. You can also parallelize requests across multiple prefixes to scale performance. However, this scaling, in the case of both read and write operations, happens gradually and is not instantaneous. While general purpose buckets are scaling to your new higher request rate, you might receive some HTTP status code 503 (Service Unavailable) errors. @@ -40 +40 @@ Prefixes organize data in a flat namespace within general purpose buckets, and t -With a hierarchical namespace, the delimiter in the object key is important. The only supported delimiter is a forward slash (`/`). Directories are determined by delimiter boundaries. For example, the object key `dir1/dir2/file1.txt` results in the directories `dir1`/ and `dir2/` being automatically created, and the object `file1.txt` being added to the `/dir2` directory in the path `dir1/dir2/file1.txt`. +#### Don't use entropy in prefixes @@ -42 +42 @@ With a hierarchical namespace, the delimiter in the object key is important. The -The directories that are created when objects are uploaded to directory buckets have no per-prefix TPS limits. Instead, each bucket can support up to 2 million TPS per S3 directory bucket. This flexibility allows your applications to parallelize read and write requests within and across directories as needed. +In Amazon S3 operations, entropy refers to the randomness in prefix naming that helps distribute workloads evenly across storage partitions. However, since directory buckets internally manage load distribution, it's not recommended to use entropy in prefixes for the best performance. This is because for directory buckets, entropy can cause requests to be slower by not reusing the directories that have already been created. @@ -44 +44 @@ The directories that are created when objects are uploaded to directory buckets -## Directory bucket horizontal scaling request parallelization +A key pattern such as `$HASH/directory/object` could end up creating many intermediate directories. In the following example, all the `job-1` s are different directories since their parents are different. Directories will be sparse and mutation and list requests will be slower. In this example there are 12 intermediate Directories that all have a single entry. @@ -46 +45,0 @@ The directories that are created when objects are uploaded to directory buckets -You can achieve the best performance by issuing multiple concurrent requests to directory buckets to spread your requests over separate connections to maximize the accessible bandwidth. S3 Express One Zone doesn't have any limits for the number of connections made to your directory bucket. Individual directories can scale performance horizontally and automatically when large numbers of concurrent writes to the same directory are happening. @@ -48 +47 @@ You can achieve the best performance by issuing multiple concurrent requests to -When an object key is initially created and its key name includes a directory, the directory is automatically created for the object. Subsequent object uploads to that same directory do not require the directory to be created, which reduces latency on object uploads to existing directories. + s3://my-bucket/0cc175b9c0f1b6a831c399e269772661/job-1/file1 @@ -50 +49,173 @@ When an object key is initially created and its key name includes a directory, t -Although both shallow and deep directory structures are supported for storing objects within a directory bucket, directory buckets do automatically scale horizontally, with lower latency on concurrent uploads to the same directory or to parallel directory siblings. + s3://my-bucket/92eb5ffee6ae2fec3ad71c777531578f/job-1/file2 + + s3://my-bucket/4a8a08f09d37b73795649038408b5f33/job-1/file3 + + s3://my-bucket/8277e0910d750195b448797616e091ad/job-1/file4 + + s3://my-bucket/e1671797c52e15f763380b45e841ec32/job-1/file5 + + s3://my-bucket/8fa14cdd754f91cc6554c9e71929cce7/job-1/file6 + +Instead, for better performance, we can remove the `$HASH` component and allow `job-1` to become a single directory, improving the density of a directory. In the following example, the single intermediate directory that has 6 entries can lead to better performance, compared with the previous example. + + + s3://my-bucket/job-1/file1 + + s3://my-bucket/job-1/file2 + + s3://my-bucket/job-1/file3 + + s3://my-bucket/job-1/file4 + + s3://my-bucket/job-1/file5 + + s3://my-bucket/job-1/file6 + +This performance advantage occurs because when an object key is initially created and its key name includes a directory, the directory is automatically created for the object. Subsequent object uploads to that same directory do not require the directory to be created, which reduces latency on object uploads to existing directories. + +#### Use a separator other than the delimiter / to separate parts of your key if you don't need the ability to logically group objects during `ListObjectsV2` calls + +Since the `/` delimiter is treated specially for directory buckets, it should be used with intention. While directory buckets do not lexicographically order objects, objects within a directory are still grouped together in `ListObjectsV2` outputs. If you don't need this functionality, you can replace `/` with another character as a separator to not cause the creation of intermediate directories. + +For example, assume the following keys are in a `YYYY/MM/DD/HH/` prefix pattern + + + s3://my-bucket/2024/04/00/01/file1 + + s3://my-bucket/2024/04/00/02/file2 + + s3://my-bucket/2024/04/00/03/file3 + + s3://my-bucket/2024/04/01/01/file4 + + s3://my-bucket/2024/04/01/02/file5 + + s3://my-bucket/2024/04/01/03/file6 + +If you don't have the need to group objects by hour or day in `ListObjectsV2` results, but you need to group objects by month, the following key pattern of `YYYY/MM/DD-HH-` will lead to significantly fewer directories and better performance for the `ListObjectsV2` operation. + + + s3://my-bucket/2024/04/00-01-file1 + + s3://my-bucket/2024/04/00-01-file2 + + s3://my-bucket/2024/04/00-01-file3 + + s3://my-bucket/2024/04/01-02-file4 + + s3://my-bucket/2024/04/01-02-file5 + + s3://my-bucket/2024/04/01-02-file6 + +#### Use delimited list operations where possible + +A `ListObjectsV2` request without a `delimiter` performs depth-first recursive traversal of all directories. A `ListObjectsV2` request with a `delimiter` retrieves only entries in the directory specified by the `prefix` parameter, reducing request latency and increasing aggregate keys per second. For directory buckets, use delimited list operations where possible. Delimited lists result in directories being visited fewer times, which leads to more keys per second and lower request latency. + +For example, for the following directories and objects in your directory bucket: + + + s3://my-bucket/2024/04/12-01-file1 + + s3://my-bucket/2024/04/12-01-file2 + + ... + + s3://my-bucket/2024/05/12-01-file1 + + s3://my-bucket/2024/05/12-01-file2 + + ... + + s3://my-bucket/2024/06/12-01-file1 + + s3://my-bucket/2024/06/12-01-file2 + + ... + + s3://my-bucket/2024/07/12-01-file1 + + s3://my-bucket/2024/07/12-01-file2 + + ... + +For better `ListObjectsV2` performance, use a delimited list to list your subdirectories and objects, if your application's logic allows for it. For example, you can run the following command for the delimited list operation, + + + aws s3api list-objects-v2 --bucket my-bucket --prefix '2024/' --delimiter '/' + +The output is the list of subdirectories. + + + { + "CommonPrefixes": [ + { + "Prefix": "2024/04/" + }, + { + "Prefix": "2024/05/" + }, + { + "Prefix": "2024/06/" + }, + { + "Prefix": "2024/07/" + } + ] + } + +To list each subdirectory with better performance, you can run a command like the following example: + +Command: + + + aws s3api list-objects-v2 --bucket my-bucket --prefix '2024/04' --delimiter '/' + +Output: + + + { + "Contents": [ + { + "Key": "2024/04/12-01-file1" + }, + { + "Key": "2024/04/12-01-file2" + }