AWS msk documentation change
Summary
Updated MSK Replicator troubleshooting guide with improved formatting, clearer headings, restructured troubleshooting steps, added troubleshooting for self-managed Kafka clusters, and consolidated failure explanations.
Security assessment
The changes include security-related troubleshooting guidance for self-managed Kafka clusters (SASL/SCRAM authentication, SSL certificates, security group configurations) and emphasize IAM role permissions and network ACL verification. However, there is no evidence these changes address a specific security vulnerability or incident; they appear to be routine documentation improvements and expansion of existing security best practices.
Diff
diff --git a/msk/latest/developerguide/msk-replicator-troubleshooting.md b/msk/latest/developerguide/msk-replicator-troubleshooting.md index e4bada45a..f8435627a 100644 --- a//msk/latest/developerguide/msk-replicator-troubleshooting.md +++ b//msk/latest/developerguide/msk-replicator-troubleshooting.md @@ -7 +7 @@ -MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in the CREATING stateMSK Replicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterMSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target clusterReplication latency is high or keeps increasingUsing ReplicatorFailure metric +Replicator state goes from CREATING to FAILEDReplicator appears stuck in the CREATING stateReplicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterReplicator is not syncing consumer group offsetsReplication latency is high or keeps increasingTroubleshooting using the ReplicatorFailure metricTroubleshoot replication from self-managed Kafka clusters @@ -9 +9 @@ MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in -# Troubleshoot MSK Replicator +# Troubleshoot Amazon MSK Replicator @@ -11 +11 @@ MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in -The following information can help you troubleshoot problems that you might have with MSK Replicator. See [Troubleshoot your Amazon MSK cluster](./troubleshooting.html) for problem solving information about other Amazon MSK features. You can also post your issue to [AWS re:Post](https://repost.aws/). +The following information can help you troubleshoot problems with MSK Replicator. See [Troubleshoot your Amazon MSK cluster](./troubleshooting.html) for other Amazon MSK features. You can also post your issue to [AWS re:Post](https://repost.aws/). @@ -13 +13 @@ The following information can help you troubleshoot problems that you might have -## MSK Replicator state goes from CREATING to FAILED +## Replicator state goes from CREATING to FAILED @@ -15 +15 @@ The following information can help you troubleshoot problems that you might have -Here are some common causes for MSK Replicator creation failure. +Common causes for MSK Replicator creation failure: @@ -17 +17 @@ Here are some common causes for MSK Replicator creation failure. - 1. Verify that the security groups you provided for the Replicator creation in the Target cluster section have outbound rules to allow traffic to your target cluster's security groups. Also verify that your target cluster's security groups have inbound rules that accept traffic from the security groups you provide for the Replicator creation in the Target cluster section. See [Choose your target cluster](./msk-replicator-create-console.html#msk-replicator-create-console-choose-target). + 1. Verify that the security groups you provided for the target cluster have outbound rules to allow traffic to your target cluster's security groups, and that your target cluster's security groups have inbound rules that accept traffic from the Replicator security groups. @@ -19 +19 @@ Here are some common causes for MSK Replicator creation failure. - 2. If you are creating Replicator for cross-region replication, verify that your source cluster has multi-VPC connectivity turned on for IAM Access Control authentication method. See [Amazon MSK multi-VPC private connectivity in a single Region](./aws-access-mult-vpc.html). Also verify that the cluster policy is setup on the source cluster so that the MSK Replicator can connect to the source cluster. See [Prepare the Amazon MSK source cluster](./msk-replicator-prepare-cluster.html). + 2. For cross-region replication, verify that your source cluster has multi-VPC connectivity turned on for IAM access control and that the cluster policy is set up on the source cluster. @@ -21 +21 @@ Here are some common causes for MSK Replicator creation failure. - 3. Verify that the IAM role that you provided during MSK Replicator creation has the permissions required to read and write to your source and target clusters. Also, verify that the IAM role has permissions to write to topics. See [Configure replicator settings and permissions](./msk-replicator-create-console.html#msk-replicator-create-settings) + 3. Verify that the IAM role provided during creation has the permissions required to read and write to your source and target clusters, including permissions to write to topics. @@ -23 +23 @@ Here are some common causes for MSK Replicator creation failure. - 4. Verify that your network ACLs are not blocking the connection between the MSK Replicator and your source and target clusters. + 4. Verify that your network ACLs are not blocking the connection between the MSK Replicator and your clusters. @@ -25 +25 @@ Here are some common causes for MSK Replicator creation failure. - 5. It's possible that source or target clusters are not fully available when the MSK Replicator tried to connect to them. This might be due to excessive load, disk usage or CPU usage, which causes the Replicator to be unable to connect to the brokers. Fix the issue with the brokers and retry Replicator creation. + 5. It is possible that source or target clusters are not fully available when the MSK Replicator tried to connect. This might be due to excessive load, disk usage, or CPU usage. Fix the issue with the brokers and retry Replicator creation. @@ -32 +32 @@ After you have performed the validations above, create the MSK Replicator again. -## MSK Replicator appears stuck in the CREATING state +## Replicator appears stuck in the CREATING state @@ -34 +34 @@ After you have performed the validations above, create the MSK Replicator again. -Sometimes MSK Replicator creation can take up to 30 minutes. Wait for 30 minutes and check the state of the Replicator again. +MSK Replicator creation can take up to 30 minutes. Wait for 30 minutes and check the state of the Replicator again. @@ -36 +36 @@ Sometimes MSK Replicator creation can take up to 30 minutes. Wait for 30 minutes -## MSK Replicator is not replicating data or replicating only partial data +## Replicator is not replicating data or replicating only partial data @@ -38 +38 @@ Sometimes MSK Replicator creation can take up to 30 minutes. Wait for 30 minutes -Follow these steps to troubleshoot data replication problems. + 1. Verify that your Replicator is not running into authentication errors using the `AuthError` metric in Amazon CloudWatch. If this metric is above 0, check the IAM role policy and ensure there are no deny permissions set for the cluster permissions. @@ -40 +40 @@ Follow these steps to troubleshoot data replication problems. - 1. Verify that your Replicator is not running into any authentication errors using the AuthError metric provided by MSK Replicator in Amazon CloudWatch. If this metric is above 0, check if the policy of the IAM role you provided for the replicator is valid and there aren't deny permissions set for the cluster permissions. Based on clusterAlias dimension, you can identify if the source or target cluster is experiencing authentication errors. + 2. Verify that your source and target clusters are not experiencing issues (too many connections, disk at full capacity, or high CPU usage). @@ -42 +42 @@ Follow these steps to troubleshoot data replication problems. - 2. Verify that your source and target clusters are not experiencing any issues. It is possible that the Replicator is not able to connect to your source or target cluster. This might happen due to too many connections, disk at full capacity or high CPU usage. + 3. Verify that your clusters are reachable using the `KafkaClusterPingSuccessCount` metric. If this metric is 0 or has no datapoint, check network and IAM role permissions. @@ -44 +44 @@ Follow these steps to troubleshoot data replication problems. - 3. Verify that your source and target clusters are reachable from MSK Replicator using the KafkaClusterPingSuccessCount metric in Amazon CloudWatch. Based on clusterAlias dimension, you can identify if the source or target cluster is experiencing auth errors. If this metric is 0 or has no datapoint, the connection is unhealthy. You should check network and IAM role permissions that MSK Replicator is using to connect to your clusters. + 4. Verify that your Replicator is not running into failures using the `ReplicatorFailure` metric. If above 0, check the IAM role for topic-level permissions. @@ -46 +46 @@ Follow these steps to troubleshoot data replication problems. - 4. Verify that your Replicator is not running into failures due to missing topic-level permissions using the ReplicatorFailure metric in Amazon CloudWatch. If this metric is above 0, check the IAM role you provided for topic-level permissions. + 5. Verify that the regular expression in the allow list matches the names of the topics you want to replicate, and that topics are not being excluded by the deny list. @@ -48,3 +48 @@ Follow these steps to troubleshoot data replication problems. - 5. Verify that the regular expression you provided in the allow list while creating the Replicator matches the names of the topics you want to replicate. Also, verify that the topics are not being excluded from replication due to a regular expression in the deny list. - - 6. Note that it may take up to 30 seconds for the Replicator to detect and create the new topics or topic partitions on the target cluster. Any messages produced to the source topic before the topic has been created on the target cluster will not be replicated if replicator starting position is latest (default). Alternatively, you can start replication from the earliest offset in the source cluster topic partitions if you want to replicate existing messages on your topics on the target cluster. See [Configure replicator settings and permissions](./msk-replicator-create-console.html#msk-replicator-create-settings). + 6. It may take up to 30 seconds for the Replicator to detect and create new topics. Messages produced before the topic is created on the target cluster will not be replicated if the starting position is _latest_ (default). @@ -57,3 +55 @@ Follow these steps to troubleshoot data replication problems. -As part of replicating data, MSK Replicator consumes messages from the source cluster and produces them to the target cluster. This can lead to messages having different offsets on your source and target clusters. However, if you have turned on consumer groups offsets syncing during Replicator creation, MSK Replicator will automatically translate the offsets while copying the metadata so that after failing over to the target cluster, your consumers can resume processing from near where they left off in the source cluster. - -## MSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target cluster +MSK Replicator consumes messages from the source cluster and produces them to the target cluster, which can lead to different offsets. If you have turned on consumer group offset syncing, MSK Replicator will automatically translate the offsets so that after failover, your consumers can resume processing from near where they left off. @@ -61 +57 @@ As part of replicating data, MSK Replicator consumes messages from the source cl -Follow these steps to troubleshoot metadata replication problems. +## Replicator is not syncing consumer group offsets @@ -63 +59 @@ Follow these steps to troubleshoot metadata replication problems. - 1. Verify that your data replication is working as expected. If not, see MSK Replicator is not replicating data or replicating only partial data. + 1. Verify that data replication is working as expected. @@ -65 +61 @@ Follow these steps to troubleshoot metadata replication problems. - 2. Verify that the regular expression you provided in the allow list while creating the Replicator matches the names of the consumer groups you want to replicate. Also, verify that the consumer groups are not being excluded from replication due to a regular expression in the deny list. + 2. Verify that the regular expression in the allow list matches the consumer groups you want to replicate. @@ -67 +63 @@ Follow these steps to troubleshoot metadata replication problems. - 3. Verify that MSK Replicator has created the topic on the target cluster. It may take up to 30 seconds for the Replicator to detect and create the new topics or topic partitions on the target cluster. Any messages produced to the source topic before the topic has been created on the target cluster will not be replicated if the replicator starting position is _latest_ (default). If your consumer group on the source cluster has only consumed the mesages that have not been replicated by MSK Replicator, the consumer group will not be replicated to the target cluster. After the topic is successfuly created on the target cluster, MSK Replicator will start replicating newly written messages on the source cluster to the target. Once your consumer group starts reading these messages from the source, MSK Replicator will automatically replicate the consumer group to the target cluster. Alternatively, you can start replication from the earliest offset in the source cluster topic partitions if you want to replicate existing messages on your topics on the target cluster. See [Configure replicator settings and permissions](./msk-replicator-create-console.html#msk-replicator-create-settings). + 3. Verify that MSK Replicator has created the topic on the target cluster. If your consumer group on the source cluster has only consumed messages that have not been replicated, the consumer group will not be replicated to the target cluster. Once your consumer group starts reading newly replicated messages, MSK Replicator will automatically replicate the consumer group. @@ -74 +70 @@ Follow these steps to troubleshoot metadata replication problems. -MSK Replicator optimizes consumer groups offset syncing for your consumers on the source cluster which are reading from a position closer to the end of the topic partition. If your consumer groups are lagging on the source cluster, you may see higher lag for those consumer groups on the target as compared to the source. This means after failover to the target cluster, your consumers will reprocess more duplicate messages. To reduce this lag, your consumers on the source cluster would need to catch up and start consuming from the tip of the stream (end of the topic partition). As your consumers catch up, MSK Replicator will automatically reduce the lag. +MSK Replicator optimizes consumer group offset syncing for consumers reading from near the end of the topic partition. If your consumer groups are lagging on the source cluster, you may see higher lag on the target. As your consumers catch up, MSK Replicator will automatically reduce the lag. @@ -78 +74 @@ MSK Replicator optimizes consumer groups offset syncing for your consumers on th -Here are some common causes for high replication latency. + 1. Verify that you have the right number of partitions. The following table shows the recommended minimum number of partitions for your desired throughput. @@ -80,3 +76 @@ Here are some common causes for high replication latency. - 1. Verify that you have the right number of partitions on your source and target MSK clusters. Having too few or too many partitions can impact performance. For guidance on choosing the number of partitions, see [Best practices for using MSK Replicator](./msk-replicator-best-practices.html). The following table shows the recommended minimum number of partitions for getting the throughput you want with MSK Replicator. - -Throughput and recommended minimum number of partitions Throughput (MB/s) | Minimum number of partitions required +Throughput and recommended minimum number of partitions Throughput (MB/s) | Minimum partitions required @@ -90,9 +84 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini - 2. Verify that you have enough read and write capacity in your source and target MSK clusters to support the replication traffic. MSK Replicator acts as a consumer for your source cluster (egress) and as a producer for your target cluster (ingress). Therefore, you should provision cluster capacity to support the replication traffic in addition to other traffic on your clusters. See [Best practices for using MSK Replicator](./msk-replicator-best-practices.html) for guidance on sizing your MSK clusters. - - 3. Replication latency might vary for MSK clusters in different source and destination AWS Region pairs, depending on how geographically far apart the clusters are from each other. For example, Replication latency is typically lower when replicating between clusters in the Europe (Ireland) and Europe (London) Regions compared to replication between clusters in the Europe (Ireland) and Asia Pacific (Sydney) Regions. - - 4. Verify that your Replicator is not getting throttled due to overly aggressive quotas set on your source or target clusters. You can use the ThrottleTime metric provided by MSK Replicator in Amazon CloudWatch to see the average time in milliseconds a request was throttled by brokers on your source/target cluster. If this metric is above 0, you should adjust Kafka quotas to reduce throttling so that Replicator can catch-up. See [Managing MSK Replicator throughput using Kafka quotas](./msk-replicator-best-practices.html#msk-replicator-manage-throughput-kafka-quotas) for information on managing Kafka quotas for the Replicator. - - 5. ReplicationLatency and MessageLag might increase when an AWS Region becomes degraded. Use the [AWS Service Health Dashboard](https://health.aws.amazon.com/health/status) to check for an MSK service event in the Region where your primary MSK cluster is located. If there's a service event, you can temporarily redirect your application reads and writes to the other Region. - - + 2. Verify that you have enough read and write capacity in your clusters. MSK Replicator acts as a consumer for your source cluster (egress) and as a producer for your target cluster (ingress). Provision cluster capacity to support replication traffic in addition to other traffic. @@ -99,0 +86 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini + 3. Replication latency varies by Region pair distance. @@ -101 +88 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini -## Troubleshooting MSK Replicator failures using ReplicatorFailure metric + 4. Verify that your Replicator is not getting throttled using the `ThrottleTime` metric. If above 0, adjust Kafka quotas. See [Managing throughput with Kafka quotas](./msk-replicator-bp-quotas.html). @@ -103 +90 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini -The ReplicatorFailure metric helps you monitor and detect replication issues in MSK Replicator. A non-zero value of this metric typically indicates replication failure issue, which might result from the following factors: + 5. Check the [AWS Service Health Dashboard](https://health.aws.amazon.com/health/status) for MSK service events in your Region. @@ -105 +91,0 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in - * message size limitations @@ -107 +92,0 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in - * timestamp range violations @@ -109 +93,0 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in - * record batch size problems @@ -110,0 +95 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in +## Troubleshooting using the ReplicatorFailure metric @@ -111,0 +97 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in +The `ReplicatorFailure` metric helps you monitor and detect replication issues. A non-zero value typically indicates a replication failure caused by message size limitations, timestamp range violations, or record batch size problems. If you have log delivery configured for your replicator, you can use the delivered log messages to identify the specific failure. For more details, see [MSK Replicator logs](./msk-replicator-logs.html). If log delivery is not configured, follow the steps below to query the replicator's status topic for error messages. @@ -112,0 +99 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in +If the `ReplicatorFailure` metric reports a non-zero value, follow these steps to troubleshoot: @@ -114,7 +101 @@ The ReplicatorFailure metric helps you monitor and detect replication issues in -If the ReplicatorFailure metric reports a non-zero value, follow these steps to troubleshoot the issue. - -###### Note - -For more information about this metric, see [MSK Replicator metrics](./msk-replicator-monitor.html#msk-replicator-metrics). - - 1. Configure a client that is able to connect to the target MSK cluster and has Apache Kafka CLI tools setup. For information about setting up the client and Kafka CLI tool, see [Connect to an Amazon MSK Provisioned cluster](./client-access.html). + 1. Configure a client that can connect to the target MSK cluster and has Apache Kafka CLI tools set up. See [Connect to an Amazon MSK Provisioned cluster](./client-access.html). @@ -124,7 +105 @@ For more information about this metric, see [MSK Replicator metrics](./msk-repli -Then, do the following: - - 1. Obtain the ARNs of MSK Replicator and target MSK cluster. - - 2. [Obtain the broker endpoints](./get-bootstrap-console.html) of the target MSK cluster. You'll use these endpoints in the following steps. - - 3. Run the following commands to export the MSK Replicator ARN and broker endpoints you obtained in the previous step. +Obtain the ARNs of the MSK Replicator and target MSK cluster, and [obtain the broker endpoints](./get-bootstrap-console.html) of the target MSK cluster. @@ -132 +107 @@ Then, do the following: -Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`BootstrapServerString`>, and <`ConsumerConfigFile`> used in the following examples with their actual values. + 3. Export the MSK Replicator ARN and broker endpoints: @@ -140,3 +113 @@ Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`Boots - 4. In your `<`path-to-your-kafka-installation`>/bin` directory, do the following: - - 1. Save the following script and name it `query-replicator-failure-message.sh`. + 4. In your `<path-to-your-kafka-installation>/bin` directory, save the following script as `query-replicator-failure-message.sh`. @@ -156 +127 @@ Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`Boots - # ./query-replicator-failure-message.sh ./query-replicator-failure-message.sh --replicator-arn <replicator-arn> --bootstrap-server <bootstrap-server> --consumer.config <consumer.config> + # ./query-replicator-failure-message.sh --replicator-arn <replicator-arn> --bootstrap-server <bootstrap-server> --consumer.config <consumer.config> @@ -219 +190 @@ Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`Boots - 2. Run this script to query the MSK Replicator failure messages. +Run this script to query the MSK Replicator failure messages: @@ -223 +194 @@ Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`Boots -This script outputs all the errors with their exception messages and affected topic-partitions. You can use this exception information to mitigate the failures as described in Common MSK Replicator failures and their solutions. Because the topic contains all the historical failure messages, start investigation using the last message. The following is an example of a failure message. +This script outputs all the errors with their exception messages and affected topic-partitions. Because the topic contains all the historical failure messages, start investigation using the last message. The following is an example of a failure message: @@ -230 +201 @@ This script outputs all the errors with their exception messages and affected to -### Common MSK Replicator failures and their solutions +###### Common failures and solutions @@ -232 +203 @@ This script outputs all the errors with their exception messages and affected to -The following list describes some of the MSK Replicator failures that you might experience and how to mitigate them. +The following describes common MSK Replicator failures and how to mitigate them. @@ -237,3 +208 @@ The following list describes some of the MSK Replicator failures that you might -###### Cause - -This failure occurs when the MSK Replicator fails to replicate data because the individual message size exceeds 10 MB. By default, MSK Replicator replicates messages up to 10 MB in size. +_Cause:_ The individual message size exceeds 10 MB (the default maximum). @@ -246,3 +215 @@ The following is an example of this failure message type. -###### Solution - -Reduce the individual message sizes in your topic. If you're unable to do so, follow these instructions for [requesting a limit increase](./limits.html#request-msk-quota-increase). +_Solution:_ Reduce the individual message sizes in your topic. If you cannot, follow the instructions for [requesting a limit increase](./limits.html#request-msk-quota-increase). @@ -253,3 +220 @@ Reduce the individual message sizes in your topic. If you're unable to do so, fo -###### Cause - -This failure occurs when the message size exceeds the target cluster’s maximum message size. +_Cause:_ The message size exceeds the target cluster's maximum message size. @@ -262,3 +227 @@ The following is an example of this failure message type. -###### Solution - -Increase the `max.message.bytes` configuration on the target cluster or corresponding target cluster topic. Set the target cluster’s `max.message.bytes` configuration to match your largest uncompressed message size. For information about doing this, see [max.message.bytes](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes). +_Solution:_ Increase the `max.message.bytes` configuration on the target cluster or topic. See [max.message.bytes](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes). @@ -269,3 +232 @@ Increase the `max.message.bytes` configuration on the target cluster or correspo -###### Cause - -This failure occurs because the individual message timestamp falls outside of the target cluster’s allowed range. +_Cause:_ The message timestamp falls outside the target cluster's allowed range. @@ -278,3 +239 @@ The following is an example of this failure message type. -###### Solution - -Update the target cluster’s `message.timestamp.before.max.ms` configuration to allow for messages with older timestamps. For information about doing this, see [message.timestamp.before.max.ms](https://kafka.apache.org/documentation/#topicconfigs_message.timestamp.before.max.ms). +_Solution:_ Update the target cluster's `message.timestamp.before.max.ms` configuration. See [message.timestamp.before.max.ms](https://kafka.apache.org/documentation/#topicconfigs_message.timestamp.before.max.ms). @@ -285,3 +244 @@ Update the target cluster’s `message.timestamp.before.max.ms` configuration to -###### Cause - -This failure occurs because the record batch size exceeds the segment size set for the topic on the target cluster. MSK Replicator supports a maximum batch size of 1 MB. +_Cause:_ The record batch size exceeds the segment size set for the topic on the target cluster. MSK Replicator supports a maximum batch size of 1 MB. @@ -294,3 +251 @@ The following is an example of this failure message type. -###### Solution