AWS glue documentation change
Summary
Major restructuring of zero-ETL integration target documentation, renaming target types from SageMaker Lakehouse to Lakehouse architecture of Amazon SageMaker, adding detailed configuration steps for general purpose S3 targets, S3 Tables targets, and Amazon Redshift Managed Storage targets, including IAM policies, RBAC policies, and CLI commands.
Security assessment
The changes primarily restructure documentation and add detailed configuration steps for different target types. While the documentation includes security-related content like IAM policies, RBAC policies, and cross-account permissions, there is no evidence of addressing a specific security vulnerability or incident. The changes add security documentation by providing detailed IAM role policies, trust policies, and resource-based access control policies for zero-ETL integrations, which helps users properly configure security permissions.
Diff
diff --git a/glue/latest/dg/zero-etl-target.md b/glue/latest/dg/zero-etl-target.md index 1001122c8..39084ca69 100644 --- a//glue/latest/dg/zero-etl-target.md +++ b//glue/latest/dg/zero-etl-target.md @@ -7 +7 @@ -SageMaker Lakehouse with S3 storageS3 Tables targetSageMaker Lakehouse with Amazon Redshift storageAmazon Redshift data warehouse targetConfiguring the integration target +General purpose S3 targetS3 Tables targetAmazon Redshift Managed Storage targetAmazon Redshift data warehouse targetConfiguring the integration target @@ -9 +9 @@ SageMaker Lakehouse with S3 storageS3 Tables targetSageMaker Lakehouse with Amaz -# Configuring a zero-ETL integration target +# Configuring a target for a zero-ETL integration @@ -11 +11 @@ SageMaker Lakehouse with S3 storageS3 Tables targetSageMaker Lakehouse with Amaz -There are several options offered by AWS when configuring a target for a zero-ETL integration. The target may be an encrypted Amazon Redshift data warehouse or an Amazon SageMaker Lakehouse catalog. +There are several options offered by AWS Glue when configuring a target for a zero-ETL integration. The target may be an encrypted Amazon Redshift data warehouse or a Lakehouse architecture of Amazon SageMaker. @@ -13 +13 @@ There are several options offered by AWS when configuring a target for a zero-ET -Before selecting the target for the zero-ETL integration, you need to configure one of the following target resources. +Before selecting the target for the zero-ETL integration, you need to configure one of the following target resources. The configuration options for a target in a zero-ETL integration include: @@ -15 +15 @@ Before selecting the target for the zero-ETL integration, you need to configure -The configuration options for a target in a zero-ETL integration include: + * A general purpose Amazon S3 bucket using the lakehouse architecture of Amazon SageMaker. See Configuring a general purpose S3 bucket target. @@ -17 +17 @@ The configuration options for a target in a zero-ETL integration include: - * An Amazon SageMaker Lakehouse catalog and database configured with regular Amazon S3 storage. See Configuring an Amazon SageMaker Lakehouse catalog with regular S3 storage. + * An Amazon S3 Tables bucket using the lakehouse architecture of Amazon SageMaker. See Configuring an Amazon S3 Tables bucket target. @@ -19,3 +19 @@ The configuration options for a target in a zero-ETL integration include: - * An Amazon SageMaker Lakehouse catalog configured with Amazon S3 Tables bucket. See Configuring Amazon S3 tables as a target. - - * An Amazon SageMaker Lakehouse catalog configured with Amazon Redshift managed storage. See Configuring an Amazon SageMaker Lakehouse catalog with Amazon Redshift managed storage. + * An Amazon Redshift Managed Storage using the lakehouse architecture of Amazon SageMaker. See Configuring an Amazon Redshift Managed Storage target. @@ -32 +30 @@ You cannot modify the target of a zero-ETL integration after creation. -## Configuring an Amazon SageMaker Lakehouse catalog with regular S3 storage +## Configuring a general purpose S3 bucket target @@ -34 +32 @@ You cannot modify the target of a zero-ETL integration after creation. -This section describes the prerequisites and setup steps for configuring a regular Amazon S3 bucket as storage for your Amazon SageMaker Lakehouse catalog target in a zero-ETL integration. +This section describes the prerequisites and setup steps for configuring a general purpose S3 bucket as storage for your target in a zero-ETL integration, using the Lakehouse architecture of Amazon SageMaker. @@ -36 +34 @@ This section describes the prerequisites and setup steps for configuring a regul -### Prerequisites for setting up an integration +Before creating a zero-ETL integration with the Lakehouse architecture of Amazon SageMaker using general purpose S3 storage, you need to complete the following setup tasks: @@ -38 +36 @@ This section describes the prerequisites and setup steps for configuring a regul -Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalog using regular S3 storage, you need to complete the following setup tasks: + * Set up an AWS Glue database @@ -40 +38 @@ Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalo - 1. Set up an AWS Glue database + * Provide Catalog RBAC policy @@ -42 +40 @@ Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalo - 2. Provide Catalog RBAC policy + * Create target IAM role @@ -44 +42 @@ Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalo - 3. Create target IAM role + * Associate target role, KMS (optional) and Connection (optional) with target resource @@ -45,0 +44 @@ Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalo + * (Optional) Configure target table properties @@ -49 +47,0 @@ Before creating a zero-ETL integration with an Amazon SageMaker Lakehouse catalo -After configuring the Amazon SageMaker Lakehouse catalog with regular Amazon S3 storage, you can proceed to Configuring the integration with your target to complete the integration setup. @@ -51 +49 @@ After configuring the Amazon SageMaker Lakehouse catalog with regular Amazon S3 -## Configuring Amazon S3 tables as a target +### Setting up an AWS Glue database @@ -53 +51 @@ After configuring the Amazon SageMaker Lakehouse catalog with regular Amazon S3 -This section describes the prerequisites and setup steps for configuring Amazon S3 Tables as a target for your zero-ETL integration. +To set up a target database in the Data Catalog with an Amazon S3 general purpose bucket location: @@ -55 +53 @@ This section describes the prerequisites and setup steps for configuring Amazon -### Prerequisites for setting up an integration + 1. In the AWS Glue console home page, select **Database** under Data Catalog. @@ -57 +55 @@ This section describes the prerequisites and setup steps for configuring Amazon -Before creating a zero-ETL integration with Amazon S3 Tables as a target, you need to complete the following setup tasks: + 2. Choose **Add database** in the top right corner. If you have already created a database, make sure that the location with Amazon S3 URI is set for the database. @@ -59 +57 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne - 1. Setup Amazon S3 tables bucket + 3. Enter a name and **Location** (Amazon S3 URI). Note that the location is required for the zero-ETL integration. Click **Create database** when done. @@ -61 +58,0 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne - 2. Provide Catalog RBAC policy @@ -63 +59,0 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne - 3. Create target IAM role @@ -65,0 +62 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne +###### Note @@ -66,0 +64 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne +The general purpose Amazon S3 bucket must be in the same region as the AWS Glue database. @@ -68 +66 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne -### Setup Amazon S3 tables bucket +For information on creating a new database in AWS Glue, see [Getting started with the Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/start-data-catalog.html). @@ -70 +68 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne - 1. Create an S3 table bucket in your account by following the instructions at [Getting started with Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-getting-started.html). +You can also use the [`create-database`](https://docs.aws.amazon.com/cli/latest/reference/glue/create-database.html) CLI to create the database in AWS Glue. Note that the `LocationUri` in `--database-input` is required. @@ -72 +70 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne - 2. Enable Analytics integrations with your S3-Table bucket by following these instructions: [Integrating AWS services with Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html#table-integration-procedures). +#### Optimizing Iceberg tables @@ -73,0 +72 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne +Once a table is created by AWS Glue in the target database, you can enable the compaction to speed up queries in Amazon Athena. For information on setting up the resources (IAM Role) for compaction, see [Table optimization prerequisites](https://docs.aws.amazon.com/glue/latest/dg/optimization-prerequisites.html). @@ -74,0 +74 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne +For more information on setting up compaction on the AWS Glue table created by the integration, see [Optimizing Iceberg tables](https://docs.aws.amazon.com/glue/latest/dg/table-optimizers.html). @@ -75,0 +76 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne +### Providing a catalog Resource Based Access (RBAC) policy @@ -77 +78 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne -### Provide Catalog RBAC Policy +For integrations that use an AWS Glue database, add the following permissions to the catalog RBAC Policy to allow for integrations between source and target. @@ -79 +80,52 @@ Before creating a zero-ETL integration with Amazon S3 Tables as a target, you ne -The following permissions must be added to the Catalog RBAC Policy to allow for integrations between source and Amazon S3 tables catalog target. +###### Note + +For cross-account integrations, both the user creating the integration role policy and catalog resource policy need to allow `glue:CreateInboundIntegration` on the resource. For same-account, either a resource policy or role policy allowing `glue:CreateInboundIntegration` on the resource is sufficient. Both scenarios do still need to allow `glue.amazonaws.com` to `glue:AuthorizeInboundIntegration`. + +You can access the **Catalog settings** under **Data Catalog**. Then provide the following permissions and fill in the missing information. + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Principal": { + "AWS": [ + "arn:aws:iam::123456789012:user/Alice" + ] + }, + "Effect": "Allow", + "Action": [ + "glue:CreateInboundIntegration" + ], + "Resource": [ + "arn:aws:glue:us-east-1:111122223333:catalog", + "arn:aws:glue:us-east-1:111122223333:database/database-name" + ], + "Condition": { + "StringLike": { + "aws:SourceArn": "arn:aws:dynamodb:us-east-1:444455556666:table/table-name" + } + } + }, + { + "Principal": { + "Service": [ + "glue.amazonaws.com" + ] + }, + "Effect": "Allow", + "Action": [ + "glue:AuthorizeInboundIntegration" + ], + "Resource": [ + "arn:aws:glue:us-east-1:111122223333:catalog", + "arn:aws:glue:us-east-1:111122223333:database/database-name" + ], + "Condition": { + "StringEquals": { + "aws:SourceArn": "arn:aws:dynamodb:us-east-1:444455556666:table/table-name" + } + } + } + ] + } @@ -81 +133,119 @@ The following permissions must be added to the Catalog RBAC Policy to allow for -Target AWS Glue Catalog resource policy needs to include Glue Service permissions to AuthorizeInboundIntegration. Additionally, CreateInboundIntegration permission is required either on the source principal creating the Integration or in the target AWS Glue resource policy. +### Creating a target IAM role + +Create a target IAM role with the following permissions and trust relationships: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Action": "s3:ListBucket", + "Resource": "arn:aws:s3:::amzn-s3-bucket", + "Effect": "Allow" + }, + { + "Action": [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject" + ], + "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/prefix/*", + "Effect": "Allow" + }, + { + "Action": [ + "glue:GetDatabase" + ], + "Resource": [ + "arn:aws:glue:us-east-1:111122223333:catalog", + "arn:aws:glue:us-east-1:111122223333:database/database-name" + ], + "Effect": "Allow" + }, + { + "Action": [ + "glue:CreateTable", + "glue:GetTable", + "glue:GetTables", + "glue:DeleteTable", + "glue:UpdateTable", + "glue:GetTableVersion", + "glue:GetTableVersions", + "glue:GetResourcePolicy" + ], + "Resource": [ + "arn:aws:glue:us-east-1:111122223333:catalog", + "arn:aws:glue:us-east-1:111122223333:database/database-name", + "arn:aws:glue:us-east-1:111122223333:table/database-name/*" + ], + "Effect": "Allow" + }, + {