AWS AmazonS3 documentation change
Summary
Expanded documentation about S3 Metadata tables, adding details about journal tables, live inventory tables, encryption configuration, storage organization, and maintenance processes. Introduced AWS managed table buckets and integration with SageMaker Lakehouse.
Security assessment
Added documentation about encryption options (SSE-S3/SSE-KMS) for metadata tables and IAM permission requirements. While these are security-related features, there's no indication they address a specific vulnerability. The changes improve security documentation by clarifying encryption controls and access management.
Diff
diff --git a/AmazonS3/latest/userguide/metadata-tables-overview.md b/AmazonS3/latest/userguide/metadata-tables-overview.md index 5c731ba89..fa2b8e9dc 100644 --- a//AmazonS3/latest/userguide/metadata-tables-overview.md +++ b//AmazonS3/latest/userguide/metadata-tables-overview.md @@ -9 +9 @@ How metadata tables work -Amazon S3 Metadata accelerates data discovery by automatically capturing metadata for the objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. These read-only tables are called _metadata tables_. As objects are added to, updated, and removed from your general purpose buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes. +Amazon S3 Metadata accelerates data discovery by automatically capturing metadata for objects in your general purpose buckets and storing it in read-only, fully managed Apache Iceberg tables that you can query. These read-only tables are called _metadata tables_. As objects are added to, updated, or removed from your general purpose buckets, S3 Metadata automatically refreshes the corresponding metadata tables to reflect the latest changes. @@ -22,2 +21,0 @@ By default, S3 Metadata provides three types of metadata: -For details about what data is stored in metadata tables, see [S3 Metadata tables schema](./metadata-tables-schema.html). - @@ -26 +24,30 @@ With S3 Metadata, you can easily find, store, and query metadata for your S3 obj -Metadata tables are stored in [S3 table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html), which provide storage that's optimized for tabular data. To easily query your metadata, you can integrate your table bucket with AWS Glue Data Catalog. After your table bucket is integrated with AWS Glue Data Catalog, you can directly query your metadata tables with query engines such as Amazon Athena, Amazon EMR, Amazon Redshift, Apache Spark, and Trino. You can also query your metadata tables with any other application that supports the Apache Iceberg format. To create dashboards from your metadata tables, use Amazon QuickSight. +For each general purpose bucket, you can create a metadata table configuration that contains two complementary metadata tables: + + * **Journal table** – By default, your metadata table configuration contains a _journal table_ , which captures events that occur for the objects in your bucket. The journal table records changes made to your data in near real time, helping you to identify new data uploaded to your bucket, track recently deleted objects, monitor lifecycle transitions, and more. The journal table records new objects and updates to your objects and their metadata (those updates that require either a `PUT` or a `DELETE` operation). + +The journal table captures metadata only for change events (such as uploads, updates, and deletes) that happen after you create your metadata table configuration. Because this table is queryable, you can audit the changes to your bucket through simple SQL queries. + +The journal table is required for each metadata table configuration. (In the initial release of S3 Metadata, the journal table was referred to as "the metadata table.") + +For more information about what data is stored in journal tables, see [S3 Metadata journal tables schema](./metadata-tables-schema.html). + +To help minimize your storage costs, you can choose to enable journal table record expiration. For more information, see [Expiring journal table records](./metadata-tables-expire-journal-table-records.html). + + * **Live inventory table** – Optionally, you can add a _live inventory table_ to your metadata table configuration. The live inventory table provides a simple, queryable inventory of all the objects and their versions in your bucket so that you can determine the latest state of your data. + +You can use the live inventory table to simplify and speed up business workflows and big data jobs by identifying objects that you want to process for various workloads. For example, you can query the live inventory table to find all objects stored in a particular storage class, all objects with certain tags, all objects that aren't encrypted with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS), and more. + +When you enable the live inventory table for your metadata table configuration, the table goes through a process known as _backfilling_ , during which Amazon S3 scans your general purpose bucket to retrieve the initial metadata for all objects that exist in the bucket. Depending on the number of objects in your bucket, this process can take minutes (minimum 15 minutes) to hours. When the backfilling process is finished, the status of your live inventory table changes from **Backfilling** to **Active**. After backfilling is completed, updates to your objects are typically reflected in the live inventory table within one hour. + +You're charged for backfilling your inventory table. If your general purpose bucket has more than one billion objects, you're also charged a monthly fee for your live inventory table. For more information, see [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/). + +For more information about what data is stored in live inventory tables, see [S3 Metadata live inventory tables schema](./metadata-tables-inventory-schema.html). + + + + +Your metadata tables are stored in an AWS managed S3 table bucket, which provides storage that's optimized for tabular data. To query your metadata, you can integrate your table bucket with Amazon SageMaker Lakehouse. This integration, which uses the AWS Glue Data Catalog and AWS Lake Formation, allows AWS analytics services to automatically discover and access your table data. + +After your table bucket is integrated with the AWS Glue Data Catalog, you can directly query your metadata tables with AWS analytics services such as Amazon Athena, Amazon EMR, and Amazon Redshift. You can also create interactive dashboards with your query data by using Amazon QuickSight. For more information about integrating your AWS managed S3 table bucket with Amazon SageMaker Lakehouse, see [Using Amazon S3 Tables with AWS analytics services](./s3-tables-integrating-aws.html). + +You can also query your metadata tables with Apache Spark, Apache Trino, and any other application that supports the Apache Iceberg format by using the AWS Glue Iceberg REST endpoint, Amazon S3 Tables Iceberg REST endpoint, or the Amazon S3 Tables Catalog for Apache Iceberg client catalog. For more information about accessing your metadata tables, see [Accessing table data](./s3-tables-access.html). @@ -32 +59 @@ For S3 Metadata pricing, see [Amazon S3 Pricing](https://aws.amazon.com/s3/prici -Metadata tables are managed by Amazon S3, and can't be modified by any IAM principal outside of Amazon S3 itself. (You can, however, delete your metadata tables.) As a result, metadata tables are read-only, which helps ensure that they correctly reflect the contents of your bucket. +Metadata tables are managed by Amazon S3, and can't be modified by any IAM principal outside of Amazon S3 itself. You can, however, delete your metadata tables. As a result, metadata tables are read-only, which helps ensure that they correctly reflect the contents of your general purpose bucket. @@ -34 +61 @@ Metadata tables are managed by Amazon S3, and can't be modified by any IAM princ -To keep your Apache Iceberg metadata tables performing at their best, Amazon S3 performs periodic maintenance activities on your tables, such as compaction and unreferenced file removal. These maintenance activities help to both minimize the cost of storing your metadata tables and optimize query performance. This table maintenance happens automatically, requiring no opt-in or ongoing management by you. However, if needed, you can configure these table maintenance activities. For more information, see [Table bucket maintenance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-table-buckets-maintenance.html). +To generate and store object metadata in AWS managed metadata tables, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata tables to reflect the latest changes to your data as long as the configuration is active on the general purpose bucket. @@ -36 +63,9 @@ To keep your Apache Iceberg metadata tables performing at their best, Amazon S3 -###### Note +Before you create a metadata table configuration, make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](./metadata-tables-permissions.html). + +###### Metadata table storage, organization, and encryption + +When you create your metadata table configuration, your metadata tables are stored in an AWS managed table bucket. All metadata table configurations in your account and in the same Region are stored in a single AWS managed table bucket. These AWS managed table buckets are named `aws-s3` and have the following Amazon Resource Name (ARN) format: + +`arn:aws:s3tables:`region`:`account_id`:bucket/aws-s3` + +For example, if your account ID is 123456789012 and your general purpose bucket is in US East (N. Virginia) (`us-east-1`), your AWS managed table bucket is also created in US East (N. Virginia) (`us-east-1`) and has the following ARN: @@ -38 +73 @@ To keep your Apache Iceberg metadata tables performing at their best, Amazon S3 -S3 Metadata is designed to continuously append to the metadata table as you make changes to your general purpose bucket. Each update creates a _snapshot_ —a new version of the metadata table. Because of the read-only nature of the metadata table, you can't delete records in the metadata table. You also can't use the snapshot expiration capability of S3 Tables to expire old snapshots of your metadata table. +`arn:aws:s3tables:`us-east-1`:123456789012:bucket/aws-s3` @@ -40 +75 @@ S3 Metadata is designed to continuously append to the metadata table as you make -To help minimize your costs, you can periodically delete your metadata table configuration and your metadata tables, and then recreate them. For more information, see [Deleting metadata table configurations](./metadata-tables-delete-configuration.html) and [Deleting metadata tables](./metadata-tables-delete-table.html). +By default, AWS managed table buckets are encrypted with server-side encryption using Amazon S3 managed keys (SSE-S3). After you create your first metadata configuration, you can set the default encryption setting for the AWS managed table bucket to use server-side encryption with AWS Key Management Service (AWS KMS) keys (SSE-KMS). For more information, see [Encryption for AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html#aws-managed-buckets-encryption) and [Specifying server-side encryption with AWS KMS keys (SSE-KMS) in table buckets](./s3-tables-kms-specify.html). @@ -42 +77 @@ To help minimize your costs, you can periodically delete your metadata table con -To generate and store object metadata in an S3 managed metadata table, you create a metadata table configuration for your general purpose bucket. Amazon S3 is designed to continuously update the metadata table to reflect the latest changes to your data as long as the configuration is active on the bucket. +Within your AWS managed table bucket, the metadata tables for your configuration are typically stored in a namespace with the following naming format: @@ -44 +79 @@ To generate and store object metadata in an S3 managed metadata table, you creat -To create a metadata table configuration, you must make sure that you have the necessary AWS Identity and Access Management (IAM) permissions to create and manage metadata tables. For more information, see [Setting up permissions for configuring metadata tables](./metadata-tables-permissions.html). You must also create or specify an S3 table bucket to store your metadata table in. This table bucket must be in the same AWS Region and account as your general purpose bucket. For more information about creating table buckets, see [Creating table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-buckets-create.html). +`b_`general-purpose-bucket-name`` @@ -48 +83,18 @@ To create a metadata table configuration, you must make sure that you have the n -S3 Metadata doesn't apply to any objects that already existed in your general purpose bucket before you created your metadata table configuration. In other words, S3 Metadata only captures metadata for change events (such as uploads, updates, and deletes) that happen after you have created your metadata table configuration. + * If your general purpose bucket name contains any periods, the periods are converted to underscores (`_`) in the namespace name. + + * If your general purpose bucket was created before March 1, 2018, its name might contain uppercase letters and underscores, and it might also be up to 255 characters long. If your bucket name has these characteristics, your metadata table namespace will have a different format. The general purpose bucket name will be prefixed with `b_`, truncated to 63 characters, converted to all lowercase, and suffixed with a hash. + + + + +Metadata tables have the following Amazon Resource Name (ARN) format: + +`arn:aws:s3tables:`region-code`:`account-id`:bucket/aws-s3/table/`metadata_table_name`` + +Journal tables have the name `journal`, and live inventory tables have the name `inventory`. + +When you create your metadata table configuration, you can choose to encrypt your AWS managed metadata tables with server-side encryption using AWS Key Management Service (AWS KMS) keys (SSE-KMS). If you choose to use SSE-KMS, you must provide a customer managed KMS key in the same Region as your general purpose bucket. You can set the encryption type for your tables only during table creation. After an AWS managed table is created, you can't change its encryption setting. To specify SSE-KMS for your metadata tables, you must have certain permissions. For more information, see [ Permissions for SSE-KMS](./metadata-tables-permissions.html#metadata-kms-permissions). + +The encryption setting for a metadata table takes precedence over the default bucket-level encryption setting. If you don't specify encryption for a table, it will inherit the default encryption setting from the bucket. + +AWS managed table buckets don't count toward your S3 Tables quotas. For more information about working with AWS managed table buckets and AWS managed tables, see [Working with AWS managed table buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-aws-managed-buckets.html). @@ -51,0 +104,13 @@ To monitor updates to your metadata table configuration, you can use AWS CloudTr +###### Metadata table maintenance and record expiration + +To keep your metadata tables performing at their best, Amazon S3 performs periodic maintenance activities on your tables, such as compaction and unreferenced file removal. These maintenance activities help to both minimize the cost of storing your metadata tables and optimize query performance. This table maintenance happens automatically, requiring no opt-in or ongoing management by you. + +###### Note + + * You can't control the expiration of journal table or inventory table snapshots. For each table, Amazon S3 stores a minimum of 1 snapshot for a maximum of 24 hours. + + * To help minimize your costs, you can configure journal table record expiration. By default, journal table records don't expire, and journal table records must be retained for a minimum of 7 days. For more information, see [Expiring journal table records](./metadata-tables-expire-journal-table-records.html). + + + + @@ -56 +121,3 @@ To monitor updates to your metadata table configuration, you can use AWS CloudTr - * [S3 Metadata tables schema](./metadata-tables-schema.html) + * [S3 Metadata journal tables schema](./metadata-tables-schema.html) + + * [S3 Metadata live inventory tables schema](./metadata-tables-inventory-schema.html) @@ -61,0 +129,2 @@ To monitor updates to your metadata table configuration, you can use AWS CloudTr + * [Troubleshooting S3 Metadata](./metadata-tables-troubleshooting.html) +