AWS redshift high security documentation change
Summary
Added detailed explanation of Iceberg table structure, metadata caching behavior, and critical security best practices for S3 permissions consistency
Security assessment
Explicitly warns about permission inconsistency risks when granting metadata access without data file access, which could lead to unauthorized data exposure or query failures. Documents security-critical caching behavior where permission revocations may be delayed by 2 minutes.
Diff
diff --git a/redshift/latest/dg/referencing-iceberg-tables.md b/redshift/latest/dg/referencing-iceberg-tables.md index fbd04b80a..369a1dfcf 100644 --- a//redshift/latest/dg/referencing-iceberg-tables.md +++ b//redshift/latest/dg/referencing-iceberg-tables.md @@ -82 +82,11 @@ The `USE` statement and `search_path` are only supported for `s3tablescatalog`. -Consider the following best practices when referencing Iceberg tables in Amazon Redshift: +An Apache Iceberg table is a single logical entity composed of multiple files: a root metadata file (metadata.json), manifest lists, manifest files, and data files (typically .parquet). The root metadata file serves as the entry point and contains references to all other files that make up the table. When you grant Amazon Redshift access to an Iceberg table, Amazon Redshift uses the root metadata file to discover and read all referenced data files. If Amazon Redshift has access to the root metadata file, it assumes and requires access to all underlying data files as well. This is consistent with Iceberg's design, where table-level access is the intended unit of authorization. + +To improve query performance, Amazon Redshift caches Iceberg metadata files (including the root metadata file, manifest lists, and manifest files) in memory. The root metadata file (metadata.json) is revalidated against Amazon S3 at a configurable interval (TTL). After the TTL expires, Amazon Redshift performs an Amazon S3 HEAD request on the root metadata file to verify that the IAM role still has access and that the file has not been modified. If the permission check fails or the file has changed, the cached entry is evicted and the metadata is re-fetched from Amazon S3. Because the root metadata file is the entry point for all table access, this revalidation serves as the permission gate for the entire table. Manifest lists and manifest files are cached without independent TTL revalidation — their access validity is derived from the root metadata permission check. This means that if you revoke Amazon S3 permissions on an Iceberg table, queries may continue to succeed for a maximum of 2 minutes while using cached metadata. + +###### Important + +Amazon S3 allows you to set permissions at the individual object level, which means it is technically possible to grant access to an Iceberg table's metadata while restricting access to some of its underlying data files. This creates a permission inconsistency that can lead to query failures or unexpected access errors in Amazon Redshift. + +Amazon Redshift validates access to the cached root metadata file periodically but does not validate or enforce consistency between metadata-level and data-file-level permissions within your Amazon S3 bucket. It is the customer's responsibility to ensure that permissions are applied consistently across all files that constitute an Iceberg table. + +To avoid this, consider the following best practices when referencing Iceberg tables in Amazon Redshift: @@ -91,0 +102,8 @@ Consider the following best practices when referencing Iceberg tables in Amazon + * Table level permissions – Grant permissions at the table level, not at the individual file level. + + * Uniform permissions – Ensure uniform access across the entire Amazon S3 path for your Iceberg table, including all metadata, manifest, and data files. + + * Avoid restrictive object-level policies – Do not set restrictive object-level policies on individual Parquet files within an Iceberg table's prefix. + + * Understand caching TTL for permission changes – When you revoke Amazon S3 permissions on an Iceberg table, queries may continue to succeed using cached root metadata for up to the configured TTL duration (default: 2 minutes). +