AWS AmazonRDS documentation change
Summary
Expanded zero-ETL integration documentation to include Amazon SageMaker lakehouse as a target alongside Amazon Redshift. Added new Step 3b for lakehouse configuration, including required IAM roles and permissions policies.
Security assessment
The change adds documentation about required IAM roles and permissions policies (including KMS encryption actions and S3 bucket policies) for integrating with SageMaker lakehouse. While this documents security-related configurations, there is no evidence of addressing a specific security vulnerability.
Diff
diff --git a/AmazonRDS/latest/AuroraUserGuide/zero-etl.setting-up.md b/AmazonRDS/latest/AuroraUserGuide/zero-etl.setting-up.md index a5c7edaad..25f240511 100644 --- a//AmazonRDS/latest/AuroraUserGuide/zero-etl.setting-up.md +++ b//AmazonRDS/latest/AuroraUserGuide/zero-etl.setting-up.md @@ -5 +5 @@ -Step 1: Create a custom DB cluster parameter groupStep 2: Select or create a source DB clusterStep 3: Create a target Amazon Redshift data warehouseSet up an integration using the AWS SDKsNext steps +Step 1: Create a custom DB cluster parameter groupStep 2: Select or create a source DB clusterStep 3a: Create a target data warehouseSet up an integration using the AWS SDKsStep 3b: Create a target Amazon SageMaker lakehouseNext steps @@ -7 +7 @@ Step 1: Create a custom DB cluster parameter groupStep 2: Select or create a sou -# Getting started with Aurora zero-ETL integrations with Amazon Redshift +# Getting started with Aurora zero-ETL integrations @@ -9 +9 @@ Step 1: Create a custom DB cluster parameter groupStep 2: Select or create a sou -Before you create a zero-ETL integration with Amazon Redshift, configure your Aurora DB cluster and your Amazon Redshift data warehouse with the required parameters and permissions. During setup, you'll complete the following steps: +Before you create a zero-ETL integration, configure your Aurora DB cluster and your data warehouse with the required parameters and permissions. During setup, you'll complete the following steps: @@ -15 +15 @@ Before you create a zero-ETL integration with Amazon Redshift, configure your Au - 3. Create a target Amazon Redshift data warehouse. + 3. Create a target data warehouse for Amazon Redshift or Create a target Amazon SageMaker lakehouse. @@ -20 +20 @@ Before you create a zero-ETL integration with Amazon Redshift, configure your Au -After you complete these tasks, continue to [Creating Aurora zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). +After you complete these tasks, continue to [Creating Aurora zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html) or [Creating Aurora zero-ETL integrations with an Amazon SageMaker lakehouse](./zero-etl.creating-smlh.html). @@ -27,0 +28,9 @@ You can have RDS complete these setup steps for you while you're creating the in +For Step 3, you can choose to create either a target data warehouse (Step 3a) or a target lakehouse (Step 3b) depending on your needs: + + * Choose a data warehouse if you need traditional data warehousing capabilities with SQL-based analytics. + + * Choose a Amazon SageMaker lakehouse if you need machine learning capabilities and want to use lakehouse features for data science and ML workflows. + + + + @@ -30 +39 @@ You can have RDS complete these setup steps for you while you're creating the in -Aurora zero-ETL integrations with Amazon Redshift require specific values for the DB cluster parameters that control replication. Specifically, Aurora MySQL requires _enhanced binlog_ (`aurora_enhanced_binlog`), and Aurora PostgreSQL requires _enhanced logical replication_ (`aurora.enhanced_logical_replication`). +Aurora zero-ETL integrations require specific values for the DB cluster parameters that control replication. Specifically, Aurora MySQL requires _enhanced binlog_ (`aurora_enhanced_binlog`), and Aurora PostgreSQL requires _enhanced logical replication_ (`aurora.enhanced_logical_replication`). @@ -76 +85 @@ If you enable or disable the `aurora.enhanced_logical_replication` DB cluster pa -After you create a custom DB cluster parameter group, choose or create an Aurora DB cluster. This cluster will be the source of data replication to Amazon Redshift. You can specify a DB cluster that uses provisioned DB instances or Aurora Serverless v2 DB instances as the source. For instructions to create a DB cluster, see [Creating an Amazon Aurora DB cluster](./Aurora.CreateInstance.html) or [Creating a DB cluster that uses Aurora Serverless v2](./aurora-serverless-v2.create.html). +After you create a custom DB cluster parameter group, choose or create an Aurora DB cluster. This cluster will be the source of data replication to the target data warehouse. You can specify a DB cluster that uses provisioned DB instances or Aurora Serverless v2 DB instances as the source. For instructions to create a DB cluster, see [Creating an Amazon Aurora DB cluster](./Aurora.CreateInstance.html) or [Creating a DB cluster that uses Aurora Serverless v2](./aurora-serverless-v2.create.html). @@ -78 +87 @@ After you create a custom DB cluster parameter group, choose or create an Aurora -The database must be running a supported DB engine version. For a list of supported versions, see [Supported Regions and Aurora DB engines for zero-ETL integrations with Amazon Redshift](./Concepts.Aurora_Fea_Regions_DB-eng.Feature.Zero-ETL.html). +The database must be running a supported DB engine version. For a list of supported versions, see [Supported Regions and Aurora DB engines for zero-ETL integrations](./Concepts.Aurora_Fea_Regions_DB-eng.Feature.Zero-ETL.html). @@ -86 +95 @@ If you associate the parameter group with the DB cluster _after_ the cluster is -## Step 3: Create a target Amazon Redshift data warehouse +## Step 3a: Create a target data warehouse @@ -88 +97 @@ If you associate the parameter group with the DB cluster _after_ the cluster is -After you create your source DB cluster, you must create and configure a target data warehouse in Amazon Redshift. The data warehouse must meet the following requirements: +After you create your source DB cluster, you must create and configure a target data warehouse. The data warehouse must meet the following requirements: @@ -128 +137 @@ After you create a data warehouse, you must configure the source Aurora DB clust -Rather than setting up each resource manually, you can run the following Python script to automatically set up the required resources for you. The code example uses the [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to create a source Amazon Aurora DB cluster and target Amazon Redshift data warehouse, each with the required parameter values. It then waits for the databases to be available before creating a zero-ETL integration between them. You can comment out different functions depending on which resources you need to set up. +Rather than setting up each resource manually, you can run the following Python script to automatically set up the required resources for you. The code example uses the [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to create a source Amazon Aurora DB cluster and target data warehouse, each with the required parameter values. It then waits for the databases to be available before creating a zero-ETL integration between them. You can comment out different functions depending on which resources you need to set up. @@ -541,0 +551,120 @@ Aurora PostgreSQL +## Step 3b: Create a target Amazon SageMaker lakehouse + +When creating a zero-ETL integration with an Amazon SageMaker lakehouse, you must target a AWS Glue catalog in AWS Lake Formation. + +### Configure permissions for the target AWS Glue catalog + +To enable zero-ETL integration for a catalog, you need to configure the following permissions: + + * AWS Lake Formation administrator role + + * Glue role for data transfer + + + + +The target creation role must be a Lake Formation administrator and requires the following permissions: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": "lakeformation:RegisterResource", + "Resource": "*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": [ + "s3:PutEncryptionConfiguration", + "iam:PassRole", + "glue:CreateCatalog", + "glue:GetCatalog", + "s3:PutBucketTagging", + "s3:PutLifecycleConfiguration", + "s3:PutBucketPolicy", + "s3:CreateBucket", + "redshift-serverless:CreateNamespace", + "s3:DeleteBucket", + "s3:PutBucketVersioning", + "redshift-serverless:CreateWorkgroup" + ], + "Resource": [ + "arn:aws:glue:*:account-id:catalog", + "arn:aws:glue:*:account-id:catalog/*", + "arn:aws:s3:::*", + "arn:aws:redshift-serverless:*:account-id:workgroup/*", + "arn:aws:redshift-serverless:*:account-id:namespace/*", + "arn:aws:iam::account-id:role/GlueDataCatalogDataTransferRole" + ] + } + ] + } + +The target creation role must have the following trust relationship: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "glue.amazonaws.com" + }, + "Action": "sts:AssumeRole" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:sts::account-id:assumed-role/Role" + }, + "Action": "sts:AssumeRole" + } + ] + } + +The Glue data transfer role (GlueDataCatalogDataTransferRole) is required for MySQL catalog operations and must have the following permissions: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DataTransferRolePolicy", + "Effect": "Allow", + "Action": [ + "kms:GenerateDataKey", + "kms:Decrypt", + "glue:GetCatalog", + "glue:GetDatabase" + ], + "Resource": [ + "*" + ] + } + ] + } + +The Glue data transfer role must have the following trust relationship: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": [ + "glue.amazonaws.com", + "redshift.amazonaws.com" + ] + }, + "Action": "sts:AssumeRole" + } + ] + } + @@ -544 +673 @@ Aurora PostgreSQL -With a source Aurora DB cluster and an Amazon Redshift target data warehouse, you can now create a zero-ETL integration and replicate data. For instructions, see [Creating Aurora zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). +With a source Aurora DB cluster and either an Amazon Redshift target data warehouse or Amazon SageMaker lakehouse, you can create a zero-ETL integration and replicate data. For instructions, see [Creating Aurora zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). @@ -554 +683 @@ Zero-ETL integrations -Creating zero-ETL integrations +Creating zero-ETL integrations with Amazon Redshift