AWS AmazonS3 documentation change
Summary
Expanded documentation about S3 Metadata tables configuration, adding details about journal/inventory tables, encryption options, record expiration, and migration guidance for pre-July 2025 configurations
Security assessment
Added documentation about SSE-KMS encryption options and required permissions for KMS keys, but no evidence of addressing specific vulnerabilities. The changes enhance security documentation by explaining encryption configuration options.
Diff
diff --git a/AmazonS3/latest/userguide/metadata-tables-create-configuration.md b/AmazonS3/latest/userguide/metadata-tables-create-configuration.md index c4636ae32..16cf7a2b3 100644 --- a//AmazonS3/latest/userguide/metadata-tables-create-configuration.md +++ b//AmazonS3/latest/userguide/metadata-tables-create-configuration.md @@ -5 +5 @@ -Create a metadata table configuration +Create a metadata table configurationEnabling inventory tables on metadata configurations created before July 15, 2025 @@ -9 +9,26 @@ Create a metadata table configuration -To generate and store Amazon S3 Metadata in a fully managed Apache Iceberg metadata table, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata table to reflect the latest changes to your data as long as the configuration is active on the bucket. Additionally, Amazon S3 continuously optimizes your metadata table to help reduce storage costs and improve analytics query performance. +To generate and store Amazon S3 Metadata in fully managed Apache Iceberg metadata tables, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata tables to reflect the latest changes to your data as long as the configuration is active on the bucket. Additionally, Amazon S3 continuously optimizes your metadata tables to help reduce storage costs and improve analytics query performance. + +For each general purpose bucket, you can create a metadata table configuration that contains two complementary metadata tables: + + * **Journal table** – By default, your metadata table configuration contains a _journal table_ , which captures events that occur for the objects in your bucket. The journal table records changes made to your data in near real time, helping you to identify new data uploaded to your bucket, track recently deleted objects, monitor lifecycle transitions, and more. The journal table records new objects and updates to your objects and their metadata (those updates that require either a `PUT` or a `DELETE` operation). + +The journal table captures metadata only for change events (such as uploads, updates, and deletes) that happen after you create your metadata table configuration. Because this table is queryable, you can audit the changes to your bucket through simple SQL queries. + +The journal table is required for each metadata table configuration. (In the initial release of S3 Metadata, the journal table was referred to as "the metadata table.") + +For more information about what data is stored in journal tables, see [S3 Metadata journal tables schema](./metadata-tables-schema.html). + +To help minimize your storage costs, you can choose to enable journal table record expiration. For more information, see [Expiring journal table records](./metadata-tables-expire-journal-table-records.html). + + * **Live inventory table** – Optionally, you can add a _live inventory table_ to your metadata table configuration. The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. + +You can use the live inventory table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the live inventory table to find all objects stored in a particular storage class, all objects with certain tags, all objects that aren't encrypted with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), and more. + +When you enable the live inventory table for your metadata table configuration, the table goes through a process known as _backfilling_ , during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your live inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the live inventory table within one hour. + +You're charged for backfilling your live inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your live inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). + +For more information about what data is stored in live inventory tables, see [S3 Metadata live inventory tables schema](./metadata-tables-inventory-schema.html). + + + @@ -13 +38,13 @@ Metadata tables have the following Amazon Resource Name (ARN) format: -`arn:aws:s3tables:`region-code`:`account-id`:bucket/`table-bucket-name`/table/`metadata_table_name`` +`arn:aws:s3tables:`region-code`:`account-id`:bucket/aws-s3/table/`metadata_table_name`` + +Journal tables have the name `journal`, and live inventory tables have the name `inventory`. + +When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket. These AWS managed table buckets are named `aws-s3` and have the following Amazon Resource Name (ARN) format: + +`arn:aws:s3tables:`region`:`account_id`:bucket/aws-s3` + +For example, if your account ID is 123456789012 and your general purpose bucket is in US East (N. Virginia) (`us-east-1`), your AWS managed table bucket is also created in US East (N. Virginia) (`us-east-1`) and has the following ARN: + +`arn:aws:s3tables:`us-east-1`:123456789012:bucket/aws-s3` + +By default, AWS managed table buckets are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). After you create your first metadata configuration, you can set the default encryption setting for the AWS managed table bucket to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Encryption for AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html#aws-managed-buckets-encryption) and [Specifying server-side encryption with AWS KMS keys (SSE-KMS) in table buckets](./s3-tables-kms-specify.html). @@ -15 +52,11 @@ Metadata tables have the following Amazon Resource Name (ARN) format: -Amazon S3 fully managed metadata tables are stored in the `aws_s3_metadata` namespace in your table bucket. For more information about namespaces in table buckets, see [Table namespaces](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-tables.html). +Within your AWS managed table bucket, the metadata tables for your configuration are typically stored in a namespace with the following naming format: + +`b_`general-purpose-bucket-name`` + +For more information about metadata table namespaces, see [How metadata tables work](./metadata-tables-overview.html#metadata-tables-how-they-work). + +When you create your metadata table configuration, you can choose to encrypt your AWS managed metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. You can set the encryption type for your tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. To specify SSE-KMS for your metadata tables, you must have certain permissions. For more information, see [ Permissions for SSE-KMS](./metadata-tables-permissions.html#metadata-kms-permissions). + +The encryption setting for a metadata table takes precedence over the default bucket-level encryption setting. If you don't specify encryption for a table, it will inherit the default encryption setting from the bucket. + +AWS managed table buckets don't count toward your S3 Tables quotas. For more information about working with AWS managed table buckets and AWS managed tables, see [Working with AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html). @@ -18,0 +66,11 @@ You can create a metadata table configuration by using the Amazon S3 console, th +###### Note + + * If you created your S3 Metadata configuration before July 15, 2025, we recommend that you delete and re-create your configuration so that you can expire journal table records and create an inventory table. For more information, see Enabling inventory tables on metadata configurations created before July 15, 2025. + + * If you've deleted your metadata table configuration and want to re-create a configuration for the same general purpose bucket, you must first manually delete the old journal and inventory tables from your AWS managed table bucket. Otherwise, creating the new metadata table configuration fails because those tables already exist. To delete your metadata tables, see [Delete a metadata table](./metadata-tables-delete-table.html#delete-metadata-table-procedure). + +Deleting a metadata table configuration deletes only the configuration. The AWS managed table bucket and your metadata tables still exist, even if you delete the metadata table configuration. + + + + @@ -23 +81 @@ Prerequisites -To create a metadata table configuration, you must first do the following: +Before you create a metadata table configuration make sure that you've met the following prerequisites: @@ -25 +83 @@ To create a metadata table configuration, you must first do the following: - * Make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](./metadata-tables-permissions.html). + * Before you create a metadata table configuration make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](./metadata-tables-permissions.html). @@ -27 +85 @@ To create a metadata table configuration, you must first do the following: - * Create an S3 table bucket to store your metadata table in. This table bucket must be in the same AWS Region and AWS account as your general purpose bucket. For more information about creating table buckets, see [Creating table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-create.html). If you're using the Amazon S3 console to create your configuration, you can do this step as part of that process. + * If you plan to query your metadata tables with Amazon Athena or another AWS query engine, make sure that you integrate your AWS managed table bucket with AWS analytics services. For more information, see [Using Amazon S3 Tables with AWS analytics services](./s3-tables-integrating-aws.html). @@ -29 +87 @@ To create a metadata table configuration, you must first do the following: - * Integrate your table bucket with AWS Glue Data Catalog so that you can directly query your metadata tables with query engines such as Amazon Athena, Amazon EMR, Amazon Redshift, Apache Spark, Apache Trino, and any other application that supports the Apache Iceberg format. For more information, see [Querying metadata tables with AWS analytics services](./metadata-tables-bucket-integration.html). +If you've already integrated an existing table bucket in this Region, your AWS managed table bucket is also automatically integrated. To determine the integration status for your table buckets in this Region, open the Amazon S3 console, and choose **Table buckets** in the left navigation pane. Under **Integration with AWS analytics services** , check the Region and whether the integration status says **Enabled**. @@ -50 +108 @@ Make sure that this general purpose bucket is an AWS Region where table buckets - 4. On the buckets' details page, choose the **Metadata** tab. + 4. On the bucket's details page, choose the **Metadata** tab. @@ -54 +112,13 @@ Make sure that this general purpose bucket is an AWS Region where table buckets - 6. On the **Create metadata configuration** page, under **Destination table bucket** , specify a table bucket to store your metadata table in. The table bucket must be in the same AWS account and AWS Region as your general purpose bucket. + 6. On the **Create metadata configuration** page, under **Journal table** , you can choose whether to encrypt your table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). By default, journal tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). + +If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. + +###### Important + +You can set the encryption type for your metadata tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. + + * To encrypt your journal table with SSE-S3 (the default), choose **Don't specify encryption type**. + + * To encrypt your journal table with SSE-KMS, choose **Specify encryption type**. Under **Encryption type** , choose **Server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS)**. Under **AWS KMS key** , either choose from your existing KMS keys, or enter your KMS key ARN. If you don't already have a KMS key, choose **Enter KMS key ARN** , and then choose **Create a KMS key**. + +Make sure that you've set up the necessary permissions for SSE-KMS. For more information, see [ Permissions for SSE-KMS](./metadata-tables-permissions.html#metadata-kms-permissions). @@ -56 +126 @@ Make sure that this general purpose bucket is an AWS Region where table buckets -If you don't already have a table bucket, choose **Create table bucket**. Table bucket names must be 3 to 63 characters and unique within your AWS account in the AWS Region that you've chosen. Valid characters are a-z, 0-9, and hyphens (-). For more information about creating a table bucket, see [ Creating table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-create.html). + 7. (Optional) By default, the records in your journal table don't expire. To help minimize the storage costs for your journal table, choose **Enabled** for **Record expiration**. @@ -58 +128 @@ If you don't already have a table bucket, choose **Create table bucket**. Table -When you create your table bucket, make sure that you integrate it with AWS Glue Data Catalog. For more information, see [Querying metadata tables with AWS analytics services](./metadata-tables-bucket-integration.html). +If you enable journal table record expiration, you can set the number of days to retain your journal table records. To set the **Days after which records expire** value, you can specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. @@ -60 +130 @@ When you create your table bucket, make sure that you integrate it with AWS Glue - 7. For **Metadata table name** , specify the name that you want your table to have. The metadata table name must be between 1 and 255 characters and unique within the `aws_s3_metadata` namespace in your table bucket. Valid characters are lowercase letters, numbers, and underscores (`_`). +Records will be expired within 24 to 48 hours after they become eligible for expiration. @@ -62 +132 @@ When you create your table bucket, make sure that you integrate it with AWS Glue - 8. Choose **Create metadata table configuration**. +###### Important @@ -63,0 +134 @@ When you create your table bucket, make sure that you integrate it with AWS Glue +After journal table records expire, they can't be recovered. @@ -64,0 +136 @@ When you create your table bucket, make sure that you integrate it with AWS Glue +Under **Journal table records will expire after the specified number of days** , select the checkbox. @@ -65,0 +138 @@ When you create your table bucket, make sure that you integrate it with AWS Glue + 8. (Optional) If you want to add an inventory table to your metadata table configuration, under **Live inventory table** , choose **Enabled** for **Configuration status**. @@ -67 +140,22 @@ When you create your table bucket, make sure that you integrate it with AWS Glue -If your metadata table configuration was successful, the ARN for your metadata table is displayed on the **Metadata** tab, along with the specified table bucket and metadata table name. +You can choose whether to encrypt your table with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). By default, inventory tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). + +If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. + +###### Important + +You can set the encryption type for your metadata tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. + + * To encrypt your inventory table with SSE-S3 (the default), choose **Don't specify encryption type**. + + * To encrypt your inventory table with SSE-KMS, choose **Specify encryption type**. Under **Encryption type** , choose **Server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS)**. Under **AWS KMS key** , either choose from your existing KMS keys, or enter your KMS key ARN. If you don't already have a KMS key, choose **Enter KMS key ARN** , and then choose **Create a KMS key**. + +Make sure that you've set up the necessary permissions for SSE-KMS. For more information, see [ Permissions for SSE-KMS](./metadata-tables-permissions.html#metadata-kms-permissions). + + 9. Choose **Create metadata table configuration**. + + + + +If your metadata table configuration was successful, the names and ARNs for your metadata tables are displayed on the **Metadata** tab, along with the name of your AWS managed table bucket and namespace. + +If you chose to enable an inventory table for your metadata table configuration, the table goes through a process known as _backfilling_ , during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the inventory table within one hour. @@ -81,3 +175 @@ To use the following example commands, replace the ``user input placeholders`` w - 1. If you don't already have a table bucket, use the following command to create a table bucket to store your metadata table in. Make sure the table bucket is in the same AWS Region as the general purpose bucket that you want to create a metadata table configuration for. - - aws s3tables create-table-bucket --name amzn-s3-demo-bucket --region us-east-2 + 1. Create a JSON file that contains your metadata table configuration, and save it (for example, `metadata-config.json`). The following is a sample configuration. @@ -85 +177 @@ To use the following example commands, replace the ``user input placeholders`` w - 2. To verify that your table bucket has been created, use the following command: +You must specify whether to enable or disable journal table record expiration. If you choose to enable record expiration, you must also specify the number of days after which your journal table records will expire. To set the `Days` value, you can specify any whole number between `7` and `2147483647`. For example, to retain your journal table records for one year, set this value to `365`. @@ -87 +179 @@ To use the following example commands, replace the ``user input placeholders`` w - aws s3tables list-table-buckets --region us-east-2 +You can optionally choose to configure an inventory table. @@ -89 +181 @@ To use the following example commands, replace the ``user input placeholders`` w - 3. Create a JSON file that contains your metadata table configuration, and save it (for example, `metadata-config.json`). The following is a sample configuration. +For both journal tables and inventory tables, you can optionally specify an encryption configuration. By default, metadata tables are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3), which you can specify by setting `SseAlgorithm` to `AES256`. @@ -91,3 +183 @@ To use the following example commands, replace the ``user input placeholders`` w -Table bucket names must be 3 to 63 characters and unique within your AWS account in the AWS Region that you've chosen. Valid characters are a-z, 0-9, and hyphens (-). For more information about creating a table bucket, see [ Creating table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-create.html). - -The metadata table name must be between 1 and 255 characters and unique within the `aws_s3_metadata` namespace in your table bucket. Valid characters are lowercase letters, numbers, and underscores (`_`). +To encrypt your metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), set `SseAlgorithm` to `aws:kms`. You must also set `KmsKeyArn` to the ARN of a customer managed KMS key in the same Region where your general purpose bucket is located. @@ -96,3 +186,15 @@ The metadata table name must be between 1 and 255 characters and unique within t - "S3TablesDestination": { - "TableBucketArn": "arn:aws:s3tables:us-east-2:111122223333:bucket/amzn-s3-demo-bucket", - "TableName": "test_metadata_table" + "JournalTableConfiguration": { + "RecordExpiration": { + "Expiration": "ENABLED", + "Days": 10 + }, + "EncryptionConfiguration": { + "SseAlgorithm": "AES256" + } + }, + "InventoryTableConfiguration": { + "ConfigurationState": "ENABLED", + "EncryptionConfiguration": { + "SseAlgorithm": "aws:kms", + "KmsKeyArn": "arn:aws:kms:us-east-2:account-id:key/key-id" + } @@ -102 +204 @@ The metadata table name must be between 1 and 255 characters and unique within t - 4. Use the following command to apply the metadata table configuration to your general purpose bucket (for example, ``amzn-s3-demo-source-bucket``): + 2. Use the following command to apply the metadata table configuration to your general purpose bucket (for example, ``amzn-s3-demo-bucket``): @@ -104,3 +206,3 @@ The metadata table name must be between 1 and 255 characters and unique within t - aws s3api create-bucket-metadata-table-configuration \ - --bucket amzn-s3-demo-source-bucket \ - --metadata-table-configuration file://./metadata-config.json \ + aws s3api create-bucket-metadata-configuration \ + --bucket amzn-s3-demo-bucket \ + --metadata-configuration file://./metadata-config.json \ @@ -109 +211 @@ The metadata table name must be between 1 and 255 characters and unique within t - 5. To verify that the configuration was created, use the following command: + 3. To verify that the configuration was created, use the following command: @@ -111,2 +213,2 @@ The metadata table name must be between 1 and 255 characters and unique within t - aws s3api get-bucket-metadata-table-configuration \