AWS msk documentation change
Summary
Updated documentation for creating an MSK Replicator via the AWS console. Changes include restructuring the workflow, adding support for self-managed Kafka clusters, clarifying subnet and security group configuration, adding new options for topic naming and consumer group offset synchronization, updating IAM role guidance to use managed policies, adding log delivery options, and general editorial improvements.
Security assessment
The changes are primarily feature updates, clarifications, and editorial improvements. They add documentation for security features like IAM role configuration (specifically referencing the AWSMSKReplicatorExecutionRole managed policy) and security group rules for network traffic control. However, there is no concrete evidence in the diff that these changes address a specific, disclosed security vulnerability, weakness, or incident. The updates to security group instructions (e.g., simplifying the note about allowing 'All Traffic' with source 0.0.0.0/0) are clarifications of existing, optional configurations rather than fixes for a security issue.
Diff
diff --git a/msk/latest/developerguide/msk-replicator-create-console.md b/msk/latest/developerguide/msk-replicator-create-console.md index 4f799b16d..075a3b582 100644 --- a//msk/latest/developerguide/msk-replicator-create-console.md +++ b//msk/latest/developerguide/msk-replicator-create-console.md @@ -7 +7 @@ -Choose your source clusterChoose your target clusterConfigure replicator settings and permissions +Replicator detailsChoose your source clusterChoose your target clusterConfigure replicator settings and permissions @@ -9 +9 @@ Choose your source clusterChoose your target clusterConfigure replicator setting -# Create replicator using the AWS console in the target cluster Region +# Create a replicator using the AWS console @@ -11 +11,3 @@ Choose your source clusterChoose your target clusterConfigure replicator setting -The following section explains the step-wise console workflow for creating a replicator. +###### Note + +The following steps focus on creating a replicator between two MSK clusters. MSK Replicator also supports replication between self-managed Apache Kafka clusters and Amazon MSK Provisioned clusters with Express brokers. If you are migrating from a self-managed Kafka deployment, see [Migrate from non-MSK Apache Kafka clusters to Amazon MSK Express brokers](./msk-replicator-migrate-external.html) and [Set up prerequisites for MSK Replicator with self-managed Apache Kafka clusters](./msk-replicator-external-prereqs.html) for the prerequisites specific to self-managed clusters. @@ -13 +15 @@ The following section explains the step-wise console workflow for creating a rep -**Replicator details** +## Replicator details @@ -32 +34 @@ The source cluster contains the data you want to copy to a target MSK cluster. -You can look up a cluster’s Region by going to **MSK Clusters** and looking at the **Cluster** details ARN. The Region name is embedded in the ARN string. In the following example ARN, `ap-southeast-2` is the cluster region. +You can look up a cluster's Region by going to **MSK Clusters** and looking at the **Cluster** details ARN. The Region name is embedded in the ARN string. In the following example ARN, `ap-southeast-2` is the cluster Region. @@ -36,3 +38 @@ You can look up a cluster’s Region by going to **MSK Clusters** and looking at - 2. Enter the ARN of your source cluster or browse to choose your source cluster. - - 3. Choose subnet(s) for your source cluster. + 2. Select **MSK cluster** as the cluster type, then enter the ARN of your source cluster or choose **Browse** to select it. @@ -40 +40 @@ You can look up a cluster’s Region by going to **MSK Clusters** and looking at -The console displays the subnets available in the source cluster’s Region for you to select. You must select a minimum of two subnets. For a same-region MSK Replicator, the subnets that you select set to access the source cluster and the subnets to access the target cluster must be in the same Availability Zone. + 3. Choose subnet(s) for your source cluster. The subnets will auto-populate based on your cluster selection. If they do not populate, or if you want to use different ones, you can select them manually. You must select a minimum of two subnets. For a same-region MSK Replicator, the subnets you select to access the source cluster and the subnets to access the target cluster must be in the same Availability Zone. @@ -42 +42 @@ The console displays the subnets available in the source cluster’s Region for - 4. Choose security group(s) for the MSK Replicator to access your source cluster. + 4. Choose security group(s) for the MSK Replicator to access your source cluster. The security groups will auto-populate based on your cluster selection. If they do not populate, or if you want to use different ones, you can select them manually. @@ -46 +46 @@ The console displays the subnets available in the source cluster’s Region for - * For same region replication (SRR), go to the Amazon EC2 console at https://console.aws.amazon.com/ec2/ and ensure that the security groups you will provide for the Replicator have outbound rules to allow traffic to your source cluster's security groups. Also, ensure that your source cluster's security groups have inbound rules that allow traffic from the Replicator security groups provided for the source. + * For same-region replication (SRR), go to the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/) and ensure that the security groups you will provide for the Replicator have outbound rules to allow traffic to your source cluster's security groups. Also, ensure that your source cluster's security groups have inbound rules that allow traffic from the Replicator security groups provided for the source. @@ -48 +48 @@ The console displays the subnets available in the source cluster’s Region for -###### To add inbound rules to your source cluster’s security group: +###### To add inbound rules to your source cluster's security group: @@ -50 +50 @@ The console displays the subnets available in the source cluster’s Region for - 1. In the AWS console, go to your source cluster’s details by selecting the the **Cluster name**. + 1. In the AWS console, go to your source cluster's details by selecting the **Cluster name**. @@ -52 +52 @@ The console displays the subnets available in the source cluster’s Region for - 2. Select the **Properties tab** , then scroll down to the **Network settings** pane to select the name of the **Security group** applied. + 2. Select the **Properties** tab, then scroll down to the **Network settings** pane to select the name of the **Security group** applied. @@ -64 +64 @@ The console displays the subnets available in the source cluster’s Region for -###### To add outbound rules to Replicator’s security group provided for the source: +###### To add outbound rules to Replicator's security group provided for the source: @@ -76 +76 @@ The console displays the subnets available in the source cluster’s Region for - 6. In the **Source** column, type the name of the MSK source cluster’s security group, and then select **Save rules**. + 6. In the **Source** column, type the name of the MSK source cluster's security group, and then select **Save rules**. @@ -83,7 +83 @@ The console displays the subnets available in the source cluster’s Region for -Alternately, if you do not want to restrict traffic using your security groups, you can add inbound and outbound rules allowing All Traffic. - -1\. Select **Add rule**. - -2\. In the **Type** column, select **All Traffic**. - -3\. In the Source column, type `0.0.0.0/0`, and then select **Save rules**. +Alternately, if you do not want to restrict traffic using your security groups, you can add inbound and outbound rules allowing All Traffic with source `0.0.0.0/0`. @@ -93 +87 @@ Alternately, if you do not want to restrict traffic using your security groups, -The target cluster is the MSK provisioned or serverless cluster to which the source data is copied. +The target cluster is the MSK Provisioned or Serverless cluster to which the source data is copied. @@ -97 +91 @@ The target cluster is the MSK provisioned or serverless cluster to which the sou -MSK Replicator creates new topics in the target cluster with an auto-generated prefix added to the topic name. For instance, MSK Replicator replicates data in “`topic`” from the source cluster to a new topic in the target cluster called `<sourceKafkaClusterAlias>.topic`. This is to distinguish topics that contain data replicated from source cluster from other topics in the target cluster and to avoid data being circularly replicated between the clusters. You can find the prefix that will be added to the topic names in the target cluster under the **sourceKafkaClusterAlias** field using `DescribeReplicator` API or the **Replicator details** page on the MSK Console. The prefix in the target cluster is `<sourceKafkaClusterAlias>`. +By default, MSK Replicator creates new topics in the target cluster with an auto-generated prefix added to the topic name (for example, `<sourceKafkaClusterAlias>.topic`). This distinguishes replicated topics from other topics in the target cluster and avoids circular replication. You can find the prefix under the **sourceKafkaClusterAlias** field using the `DescribeReplicator` API or the **Replicator details** page on the MSK console. Alternatively, you can use Identical topic name replication. See [Topic naming (Prefixed vs Identical)](./msk-replicator-topic-naming.html). @@ -101 +95 @@ MSK Replicator creates new topics in the target cluster with an auto-generated p - 2. Enter the ARN of your target cluster or browse to choose your target cluster. + 2. Select **MSK cluster** as the cluster type, then enter the ARN of your target cluster or choose **Browse** to select it. @@ -103 +97 @@ MSK Replicator creates new topics in the target cluster with an auto-generated p - 3. Choose subnet(s) for your target cluster. + 3. Choose subnet(s) for your target cluster. The subnets will auto-populate based on your cluster selection. If they do not populate, or if you want to use different ones, you can select them manually. Select a minimum of two subnets. @@ -105 +99 @@ MSK Replicator creates new topics in the target cluster with an auto-generated p -The console displays subnets available in the target cluster’s Region for you to select. Select a minimum of two subnets. + 4. Choose security group(s) for the MSK Replicator to access your target cluster. The security groups will auto-populate based on your cluster selection. If they do not populate, or if you want to use different ones, you can select them manually. For more information about using security groups, see [Control traffic to your AWS resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) in the _Amazon VPC User Guide_. @@ -107 +101 @@ The console displays subnets available in the target cluster’s Region for you - 4. Choose security group(s) for the MSK Replicator to access your target cluster. +For both CRR and SRR, go to the Amazon EC2 console and ensure that the security groups you will provide to the Replicator have outbound rules to allow traffic to your target cluster's security groups. Also ensure that your target cluster's security groups have inbound rules that accept traffic from the Replicator security groups provided for the target. @@ -109 +103 @@ The console displays subnets available in the target cluster’s Region for you -The security groups available in the target cluster’s Region are displayed for you to select. The chosen security group is associated with each connection. For more information about using security groups, see the [Control traffic to your AWS resources using security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) in the _Amazon VPC User Guide_. +###### To add inbound rules to your target cluster's security group: @@ -111,5 +105 @@ The security groups available in the target cluster’s Region are displayed for - * For both cross region replication (CRR) and same region replication (SRR), go to the Amazon EC2 console at [https://console.aws.amazon.com/ec2/](https://console.aws.amazon.com/ec2/) and ensure that the security groups you will provide to the Replicator have outbound rules to allow traffic to your target cluster's security groups. Also ensure that your target cluster's security groups have inbound rules that accept traffic from the Replicator security groups provided for the target. - -###### To add inbound rules to your target cluster’s security group: - - 1. In the AWS console, go to your target cluster’s details by selecting the the **Cluster name**. + 1. In the AWS console, go to your target cluster's details by selecting the **Cluster name**. @@ -127 +117 @@ The security groups available in the target cluster’s Region are displayed for - 7. In the **Source** column, type the name of the security group that you will provide during Replicator creation for the target cluster (this may be the same as the MSK target cluster's security group), and then select **Save rules**. + 7. In the **Source** column, type the name of the security group that you will provide during Replicator creation for the target cluster, and then select **Save rules**. @@ -129 +119 @@ The security groups available in the target cluster’s Region are displayed for -###### To add outbound rules to Replicator’s security group provided for the target: +###### To add outbound rules to Replicator's security group provided for the target: @@ -141 +131 @@ The security groups available in the target cluster’s Region are displayed for - 6. In the **Port range** column, type `9098`. MSK Replicator uses IAM access control to connect to your cluster which uses port 9098. + 6. In the **Port range** column, type `9098`. @@ -143 +133 @@ The security groups available in the target cluster’s Region are displayed for - 7. In the **Source** column, type the name of the MSK target cluster’s security group, and then select **Save rules**. + 7. In the **Source** column, type the name of the MSK target cluster's security group, and then select **Save rules**. @@ -150,7 +140 @@ The security groups available in the target cluster’s Region are displayed for -Alternately, if you do not want to restrict traffic using your security groups, you can add inbound and outbound rules allowing All Traffic. - -1\. Select **Add rule**. - -2\. In the **Type** column, select **All Traffic**. - -3\. In the Source column, type `0.0.0.0/0`, and then select **Save rules**. +Alternately, if you do not want to restrict traffic using your security groups, you can add inbound and outbound rules allowing All Traffic with source `0.0.0.0/0`. @@ -164 +148 @@ Alternately, if you do not want to restrict traffic using your security groups, -MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, we recommend that you create a separate Replicator. Go to the AWS console Support Center and [create a support case](https://console.aws.amazon.com/support/home#/) if you need support for more than 750 topics per Replicator. You can monitor the number of topics being replicated using the "TopicCount" metric. See [Amazon MSK Standard broker quota](./limits.html#msk-provisioned-quota). +MSK Replicator only replicates up to 750 topics in sorted order. If you need to replicate more topics, create a separate Replicator. Go to the AWS console Support Center and [create a support case](https://console.aws.amazon.com/support/home#/) if you need support for more than 750 topics per Replicator. @@ -166 +150 @@ MSK Replicator only replicates up to 750 topics in sorted order. If you need to - 2. By default, MSK Replicator starts replication from the _latest_ (most recent) offset in the selected topics. Alternatively, you can start replication from the _earliest_ (oldest) offset in the selected topics if you want to replicate existing data on your topics. Once the Replicator is created, you can’t change this setting. This setting corresponds to the [`startingPosition`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators-replicatorarn.html#v1-replicators-replicatorarn-model-replicationstartingposition) field in the [`CreateReplicator`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators.html#CreateReplicator) request and [`DescribeReplicator`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators-replicatorarn.html#DescribeReplicator) response APIs. + 2. By default, MSK Replicator starts replication from the _latest_ (most recent) offset. Alternatively, you can start replication from the _earliest_ (oldest) offset if you want to replicate existing data. Once the Replicator is created, you cannot change this setting. This setting corresponds to the [`startingPosition`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators-replicatorarn.html#v1-replicators-replicatorarn-model-replicationstartingposition) field in the [`CreateReplicator`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators.html#CreateReplicator) request and [`DescribeReplicator`](https://docs.aws.amazon.com/msk/1.0/apireference-replicator/v1-replicators-replicatorarn.html#DescribeReplicator) response APIs. @@ -170,5 +154 @@ MSK Replicator only replicates up to 750 topics in sorted order. If you need to - * `PREFIXED` topic name replication (**Add prefix to topics name** in console): The default setting. MSK Replicator replicates “topic1” from the source cluster to a new topic in the target cluster with the name `<sourceKafkaClusterAlias>.topic1`. - - * Identical topic name replication (**Keep the same topics name** in console): Topics from the source cluster are replicated with identical topic names in the target cluster. - -This setting corresponds to the `TopicNameConfiguration` field in the `CreateReplicator` request and `DescribeReplicator` response APIs. See [How Amazon MSK Replicator works](./msk-replicator-how-it-works.html). + * `PREFIXED` topic name replication (**Add prefix to topics name** in console): The default setting. @@ -176,3 +156 @@ This setting corresponds to the `TopicNameConfiguration` field in the `CreateRep -###### Note - -By default, MSK Replicator creates new topics in the target cluster with an auto-generated prefix added to the topic name. This is to distinguish topics that contain data replicated from source cluster from other topics in the target cluster and to avoid data being circularly replicated between the clusters. Alternatively, you can create a MSK Replicator with Identical topic name replication (**Keep the same topics name** in console) so that topic names are preserved during replication. This configuration reduces the need for you to reconfigure client applications during setup and makes it simpler to operate multi-cluster streaming architectures. + * Identical topic name replication (**Keep the same topics name** in console): Topics are replicated with identical names in the target cluster. @@ -180 +158 @@ By default, MSK Replicator creates new topics in the target cluster with an auto - 4. By default, MSK Replicator copies all metadata including topic configurations, Access Control Lists (ACLs) and consumer group offsets for seamless failover. If you are not creating the Replicator for failover, you can optionally choose to turn off one or more of these settings available in the **Additional settings** section. +For more information, see [Topic naming (Prefixed vs Identical)](./msk-replicator-topic-naming.html). @@ -182,3 +160 @@ By default, MSK Replicator creates new topics in the target cluster with an auto -###### Note - -MSK Replicator does not replicate write ACLs since your producers should not be writing directly to the replicated topic in the target cluster. Your producers should write to the local topic in the target cluster after failover. See [Perform a planned failover to the secondary AWS Region](./msk-replicator-perform-planned-failover.html) for details. + 4. By default, MSK Replicator copies all metadata including topic configurations, ACLs, and consumer group offsets for seamless failover. If you are not creating the Replicator for failover, you can optionally turn off one or more of these settings in the **Additional settings** section. @@ -188,3 +164 @@ MSK Replicator does not replicate write ACLs since your producers should not be - 6. In the **Compression** pane, you can optionally choose to compress the data written to the target cluster. If you’re going to use compression, we recommend that you use the same compression method as the data in your source cluster. - - 7. In the **Access permissions** pane do either of the following: +You can also configure the **Consumer group offset sync mode** : @@ -192 +166 @@ MSK Replicator does not replicate write ACLs since your producers should not be - 1. Select **Create or update IAM role with required policies**. MSK console will automatically attach the necessary permissions and trust policy to the service execution role required to read and write to your source and target MSK clusters. + * **Legacy** (default) — Offsets are synchronized when producers write to the source cluster (unidirectional). @@ -194 +168 @@ MSK Replicator does not replicate write ACLs since your producers should not be - + * **Enhanced** — Consumer offsets are synchronized regardless of producer location (bidirectional). Requires a corresponding Replicator that replicates data from the target cluster back to the source cluster. Use this mode when setting up bidirectional replication for migration or active-active architectures. For more information, see [Consumer group offset synchronization](./msk-replicator-bidirectional-offset-sync.html). @@ -196 +170 @@ MSK Replicator does not replicate write ACLs since your producers should not be - 2. Provide your own IAM role by selecting **Choose from IAM roles that Amazon MSK can assume**. We recommend that you attach the `AWSMSKReplicatorExecutionRole` managed IAM policy to your service execution role, instead of writing your own IAM policy. + 6. In the **Compression** pane, you can optionally choose to compress the data written to the target cluster. If you use compression, we recommend using the same compression method as the data in your source cluster. @@ -198 +172 @@ MSK Replicator does not replicate write ACLs since your producers should not be - 1. Create the IAM role that the Replicator will use to read and write to your source and target MSK clusters with the below JSON as part of the trust policy and the `AWSMSKReplicatorExecutionRole` attached to the role. In the trust policy, replace the placeholder <yourAccountID> with your actual account ID. + 7. In the **Access permissions** pane, do either of the following: @@ -200 +174 @@ MSK Replicator does not replicate write ACLs since your producers should not be -JSON + 1. Select **Create or update IAM role with required policies**. The MSK console will automatically attach the necessary permissions and trust policy to the service execution role. @@ -201,0 +176 @@ JSON + @@ -203,2 +178 @@ JSON -**** - + 2. Provide your own IAM role by selecting **Choose from IAM roles that Amazon MSK can assume**. We recommend attaching the [`AWSMSKReplicatorExecutionRole`](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-AWSMSKReplicatorExecutionRole.html) managed IAM policy to your service execution role. See [Service execution role (SER)](./msk-replicator-ser.html). @@ -206,17 +180 @@ JSON - { - "Version":"2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "kafka.amazonaws.com" - }, - "Action": "sts:AssumeRole", - "Condition": { - "StringEquals": { - "aws:SourceAccount": "<yourAccountID>" - } - } - } - ] - } + 8. In the **Log delivery** pane, you can optionally configure log delivery to capture and route replication logs to your chosen destinations. By default, log delivery is not enabled. You can enable one or more of the following destinations: @@ -223,0 +182 @@ JSON + * **Deliver to Amazon CloudWatch Logs** — Analyze, query, and set alarms on the logs. @@ -225 +184 @@ JSON - 8. In the **Replicator tags** pane, you can optionally assign tags to the MSK Replicator resource. For more information, see [Tag an Amazon MSK cluster](./msk-tagging.html). For a cross-region MSK Replicator, tags are synced to the remote Region automatically when the Replicator is created. If you change tags after the Replicator is created, the change is not automatically synced to the remote Region, so you’ll need to sync local replicator and remote replicator references manually. + * **Deliver to Amazon S3** — Store and retrieve raw logs in object storage. @@ -227 +186 @@ JSON - 9. Select **Create**. + * **Deliver to Amazon Data Firehose** — Capture, transform, and deliver logs to Amazon OpenSearch Service or other Amazon Data Firehose destinations. @@ -228,0 +188 @@ JSON +For more information, see [MSK Replicator logs](./msk-replicator-logs.html). @@ -229,0 +190 @@ JSON + 9. In the **Replicator tags** pane, you can optionally assign tags to the MSK Replicator resource. For a cross-region MSK Replicator, tags are synced to the remote Region automatically when the Replicator is created. @@ -230,0 +192 @@ JSON + 10. Select **Create**. @@ -232 +193,0 @@ JSON -If you want to restrict `kafka-cluster:WriteData` permission, refer to the _Create authorization policies_ section of [How IAM access control for Amazon MSK works](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#how-to-use-iam-access-control). You'll need to add `kafka-cluster:WriteDataIdempotently` permission to both the source and target cluster. @@ -234 +194,0 @@ If you want to restrict `kafka-cluster:WriteData` permission, refer to the _Crea -It takes approximately 30 minutes for the MSK Replicator to be successfully created and transitioned to RUNNING status. @@ -236 +195,0 @@ It takes approximately 30 minutes for the MSK Replicator to be successfully crea -If you create a new MSK Replicator to replace one that you deleted, the new Replicator starts replication from the latest offset. @@ -238 +197 @@ If you create a new MSK Replicator to replace one that you deleted, the new Repl