AWS AmazonRDS documentation change
Summary
General documentation updates to broaden zero-ETL integration scope beyond Amazon Redshift, including title changes and modified step descriptions.
Security assessment
Structural and naming changes to support new integration target (SageMaker lakehouse) without security-specific content
Diff
diff --git a/AmazonRDS/latest/UserGuide/zero-etl.setting-up.md b/AmazonRDS/latest/UserGuide/zero-etl.setting-up.md index 98a708e34..dc0b37796 100644 --- a//AmazonRDS/latest/UserGuide/zero-etl.setting-up.md +++ b//AmazonRDS/latest/UserGuide/zero-etl.setting-up.md @@ -5 +5 @@ -Step 1: Create a custom DB parameter groupStep 2: Select or create a source databaseStep 3: Create a target Amazon Redshift data warehouseSet up an integration using the AWS SDKsNext steps +Step 1: Create a custom DB parameter groupStep 2: Select or create a source databaseStep 3a: Create a target data warehouseSet up an integration using the AWS SDKsStep 3b: Create a target Amazon SageMaker lakehouseNext steps @@ -7 +7 @@ Step 1: Create a custom DB parameter groupStep 2: Select or create a source data -# Getting started with Amazon RDS zero-ETL integrations with Amazon Redshift +# Getting started with Amazon RDS zero-ETL integrations @@ -9 +9 @@ Step 1: Create a custom DB parameter groupStep 2: Select or create a source data -Before you create a zero-ETL integration with Amazon Redshift, configure your RDS database and your Amazon Redshift data warehouse with the required parameters and permissions. During setup, you'll complete the following steps: +Before you create a zero-ETL integration, configure your RDS database and your data warehouse with the required parameters and permissions. During setup, you'll complete the following steps: @@ -15 +15 @@ Before you create a zero-ETL integration with Amazon Redshift, configure your RD - 3. Create a target Amazon Redshift data warehouse. + 3. Create a target data warehouse for Amazon Redshift or Create a target Amazon SageMaker lakehouse. @@ -20 +20 @@ Before you create a zero-ETL integration with Amazon Redshift, configure your RD -After you complete these tasks, continue to [Creating Amazon RDS zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). +After you complete these tasks, continue to [Creating Amazon RDS zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html) or [Creating Amazon RDS zero-ETL integrations with an Amazon SageMaker lakehouse](./zero-etl.creating-smlh.html). @@ -25,0 +26,9 @@ You can have RDS complete these setup steps for you while you're creating the in +For Step 3, you can choose to create either a target data warehouse (Step 3a) or a target lakehouse (Step 3b) depending on your needs: + + * Choose a data warehouse if you need traditional data warehousing capabilities with SQL-based analytics. + + * Choose a Amazon SageMaker lakehouse if you need machine learning capabilities and want to use lakehouse features for data science and ML workflows. + + + + @@ -28 +37 @@ You can have RDS complete these setup steps for you while you're creating the in -Amazon RDS zero-ETL integrations with Amazon Redshift require specific values for the DB parameters that control binary logging (binlog). To configure binary logging, you must first create a custom DB parameter group, and then associate it with the source database. Configure the following parameter values. For instructions to create a parameter group, see [DB parameter groups for Amazon RDS DB instances](./USER_WorkingWithDBInstanceParamGroups.html). We recommend that you configure all parameter values within the same request to avoid dependency issues. +Amazon RDS zero-ETL integrations require specific values for the DB parameters that control binary logging (binlog). To configure binary logging, you must first create a custom DB parameter group, and then associate it with the source database. Configure the following parameter values. For instructions to create a parameter group, see [DB parameter groups for Amazon RDS DB instances](./USER_WorkingWithDBInstanceParamGroups.html). We recommend that you configure all parameter values within the same request to avoid dependency issues. @@ -41 +50 @@ In addition, make sure that the `binlog_row_value_options` parameter is _not_ se -After you create a custom DB parameter group, choose or create an RDS for MySQL database. This database will be the source of data replication to Amazon Redshift. For instructions to create a Single-AZ or Multi-AZ DB instance, see [Creating an Amazon RDS DB instance](./USER_CreateDBInstance.html). For instructions to create a Multi-AZ DB cluster, see [Creating a Multi-AZ DB cluster for Amazon RDS](./create-multi-az-db-cluster.html). +After you create a custom DB parameter group, choose or create an RDS for MySQL database. This database will be the source of data replication to the target data warehouse. For instructions to create a Single-AZ or Multi-AZ DB instance, see [Creating an Amazon RDS DB instance](./USER_CreateDBInstance.html). For instructions to create a Multi-AZ DB cluster, see [Creating a Multi-AZ DB cluster for Amazon RDS](./create-multi-az-db-cluster.html). @@ -43 +52 @@ After you create a custom DB parameter group, choose or create an RDS for MySQL -The database must be running a supported DB engine version. For a list of supported versions, see [Supported Regions and DB engines for Amazon RDS zero-ETL integrations with Amazon Redshift](./Concepts.RDS_Fea_Regions_DB-eng.Feature.ZeroETL.html). +The database must be running a supported DB engine version. For a list of supported versions, see [Supported Regions and DB engines for Amazon RDS zero-ETL integrations](./Concepts.RDS_Fea_Regions_DB-eng.Feature.ZeroETL.html). @@ -53 +62 @@ In addition, make sure that automated backups are enabled on the database. For m -## Step 3: Create a target Amazon Redshift data warehouse +## Step 3a: Create a target data warehouse @@ -55 +64 @@ In addition, make sure that automated backups are enabled on the database. For m -After you create your source database, you must create and configure a target data warehouse in Amazon Redshift. The data warehouse must meet the following requirements: +After you create your source database, you must create and configure a target data warehouse. The data warehouse must meet the following requirements: @@ -95 +104 @@ After you create a data warehouse, you must configure the source RDS database as -Rather than setting up each resource manually, you can run the following Python script to automatically set up the required resources for you. The code example uses the [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to create a source RDS for MySQL DB instance and target Amazon Redshift data warehouse, each with the required parameter values. It then waits for the databases to be available before creating a zero-ETL integration between them. You can comment out different functions depending on which resources you need to set up. +Rather than setting up each resource manually, you can run the following Python script to automatically set up the required resources for you. The code example uses the [AWS SDK for Python (Boto3)](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) to create a source RDS for MySQL DB instance and target data warehouse, each with the required parameter values. It then waits for the databases to be available before creating a zero-ETL integration between them. You can comment out different functions depending on which resources you need to set up. @@ -278,0 +288,120 @@ Within the script, optionally modify the names of the source, target, and parame +## Step 3b: Create a target Amazon SageMaker lakehouse + +When creating a zero-ETL integration with an Amazon SageMaker lakehouse, you must target a AWS Glue catalog in AWS Lake Formation. + +### Configure permissions for the target AWS Glue catalog + +To enable zero-ETL integration for a catalog, you need to configure the following permissions: + + * AWS Lake Formation administrator role + + * Glue role for data transfer + + + + +The target creation role must be a Lake Formation administrator and requires the following permissions: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": "lakeformation:RegisterResource", + "Resource": "*" + }, + { + "Sid": "VisualEditor1", + "Effect": "Allow", + "Action": [ + "s3:PutEncryptionConfiguration", + "iam:PassRole", + "glue:CreateCatalog", + "glue:GetCatalog", + "s3:PutBucketTagging", + "s3:PutLifecycleConfiguration", + "s3:PutBucketPolicy", + "s3:CreateBucket", + "redshift-serverless:CreateNamespace", + "s3:DeleteBucket", + "s3:PutBucketVersioning", + "redshift-serverless:CreateWorkgroup" + ], + "Resource": [ + "arn:aws:glue:*:account-id:catalog", + "arn:aws:glue:*:account-id:catalog/*", + "arn:aws:s3:::*", + "arn:aws:redshift-serverless:*:account-id:workgroup/*", + "arn:aws:redshift-serverless:*:account-id:namespace/*", + "arn:aws:iam::account-id:role/GlueDataCatalogDataTransferRole" + ] + } + ] + } + +The target creation role must have the following trust relationship: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": "glue.amazonaws.com" + }, + "Action": "sts:AssumeRole" + }, + { + "Effect": "Allow", + "Principal": { + "AWS": "arn:aws:sts::account-id:assumed-role/Role" + }, + "Action": "sts:AssumeRole" + } + ] + } + +The Glue data transfer role (GlueDataCatalogDataTransferRole) is required for MySQL catalog operations and must have the following permissions: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DataTransferRolePolicy", + "Effect": "Allow", + "Action": [ + "kms:GenerateDataKey", + "kms:Decrypt", + "glue:GetCatalog", + "glue:GetDatabase" + ], + "Resource": [ + "*" + ] + } + ] + } + +The Glue data transfer role must have the following trust relationship: + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Service": [ + "glue.amazonaws.com", + "redshift.amazonaws.com" + ] + }, + "Action": "sts:AssumeRole" + } + ] + } + @@ -281 +410 @@ Within the script, optionally modify the names of the source, target, and parame -With a source RDS database and an Amazon Redshift target data warehouse, you can now create a zero-ETL integration and replicate data. For instructions, see [Creating Amazon RDS zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). +With a source RDS database and either an Amazon Redshift target data warehouse or Amazon SageMaker lakehouse, you can create a zero-ETL integration and replicate data. For instructions, see [Creating Amazon RDS zero-ETL integrations with Amazon Redshift](./zero-etl.creating.html). @@ -291 +420 @@ Zero-ETL integrations -Creating zero-ETL integrations +Creating zero-ETL integrations with Amazon Redshift