AWS msk documentation change
Summary
Updated troubleshooting guide for MSK Replicator failures with improved error categorization, script enhancements for failure detection, and detailed mitigation steps for common replication issues
Security assessment
The changes focus on improving troubleshooting procedures and script reliability without mentioning security vulnerabilities, patches, or security incidents. Enhancements include better error handling in scripts and clearer documentation for operational issues.
Diff
diff --git a/msk/latest/developerguide/msk-replicator-troubleshooting.md b/msk/latest/developerguide/msk-replicator-troubleshooting.md index 007fb803d..0918197ee 100644 --- a//msk/latest/developerguide/msk-replicator-troubleshooting.md +++ b//msk/latest/developerguide/msk-replicator-troubleshooting.md @@ -5 +5 @@ -MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in the CREATING stateMSK Replicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterMSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target clusterReplication latency is high or keeps increasingMSK Replicator failure +MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in the CREATING stateMSK Replicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterMSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target clusterReplication latency is high or keeps increasingUsing ReplicatorFailure metric @@ -99 +99 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini -## MSK Replicator failed to replicate data +## Troubleshooting MSK Replicator failures using ReplicatorFailure metric @@ -101 +101,16 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini -When data replication issues occur, the ReplicatorFailure metric helps you identify and resolve those problems. When this metric emits a non-zero value, you might experience replication failures related to message size limitations, timestamp range violations, record batch size problems, among others. To restore data replication across your MSK clusters based on the type of error that occurs, perform the following steps. +The ReplicatorFailure metric helps you monitor and detect replication issues in MSK Replicator. A non-zero value of this metric typically indicates replication failure issue, which might result from the following factors: + + * message size limitations + + * timestamp range violations + + * record batch size problems + + + + +If the ReplicatorFailure metric reports a non-zero value, follow these steps to troubleshoot the issue. + +###### Note + +For more information about this metric, see [MSK Replicator metrics](./msk-replicator-monitor.html#msk-replicator-metrics). @@ -115 +130 @@ Then, do the following: -Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with the ARN and broker endpoint that you obtained in the previous step. +Make sure that you replace the placeholder values for <`ReplicatorARN`>, <`BootstrapServerString`>, and <`ConsumerConfigFile`> used in the following examples with their actual values. @@ -120,0 +136,2 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with + export CONSUMER_CONFIG_FILE=<ConsumerConfigFile> + @@ -135 +152 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - # + # --consumer.config: Consumer config properties file @@ -137 +154 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - # ./query-replicator-failure-message.sh --replicator-arn <arn> --bootstrap-server <bootstrap-server> + # ./query-replicator-failure-message.sh ./query-replicator-failure-message.sh --replicator-arn <replicator-arn> --bootstrap-server <bootstrap-server> --consumer.config <consumer.config> @@ -140 +157 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - echo "USAGE: $0 --replicator-arn <arn> --bootstrap-server <bootstrap-server>" + echo "USAGE: $0 ./query-replicator-failure-message.sh --replicator-arn <replicator-arn> --bootstrap-server <bootstrap-server> --consumer.config <consumer.config>" @@ -142,0 +160,2 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with + echo "--consumer.config <String: config file> REQUIRED: Consumer config properties file." + exit 1 @@ -145,5 +164 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - if [ $# -eq 2 ]; - then - print_usage - exit 1 - fi + # Initialize variables @@ -151,0 +167,3 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with + consumer_config="" + + # Parse arguments @@ -155,3 +173,5 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - replicator_arn="$2" - shift 2 - ;; + if [ -z "$2" ]; then + echo "Error: --replicator-arn requires an argument." + print_usage + fi + replicator_arn="$2"; shift 2 ;; @@ -159,5 +179,2 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - bootstrap_server="$2" - shift 2 - ;; - *) - echo "Unknown option: $1" + if [ -z "$2" ]; then + echo "Error: --bootstrap-server requires an argument." @@ -165,2 +182,9 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - exit 1 - ;; + fi + bootstrap_server="$2"; shift 2 ;; + --consumer.config) + if [ -z "$2" ]; then + echo "Error: --consumer.config requires an argument." + print_usage + fi + consumer_config="$2"; shift 2 ;; + *) echo "Unknown option: $1"; print_usage ;; @@ -169,0 +194,7 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with + # Check for required arguments + if [ -z "$replicator_arn" ] || [ -z "$bootstrap_server" ] || [ -z "$consumer_config" ]; then + echo "Error: --replicator-arn, --bootstrap-server, and --consumer.config are required." + print_usage + fi + + # Extract replicator name and suffix from ARN @@ -172,0 +204,2 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with + + # List topics and find the status topic @@ -176,3 +209,2 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - # Check if any topic matches the MSK Replicator status topic name - echo "$topics" | grep -F "$status_topic_name" > /dev/null - if [ $? -eq 0 ]; then + # Check if the status topic exists + if echo "$topics" | grep -Fq "$status_topic_name"; then @@ -180 +212 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - ./kafka-console-consumer.sh --bootstrap-server $bootstrap_server --consumer.config client.properties --topic "$status_topic_name" --from-beginning --property print.key=true --property key.separator=":" | grep "Exception" | sed -n 's/.*Exception:\(.*\) Topic: \([^,]*\), Partition: \([^\]*\).*/ReplicatorException:\1 Topic: \2, Partition: \3/p' + ./kafka-console-consumer.sh --bootstrap-server "$bootstrap_server" --consumer.config "$consumer_config" --topic "$status_topic_name" --from-beginning | stdbuf -oL grep "Exception" | stdbuf -oL sed -n 's/.*Exception:\(.*\) Topic: \([^,]*\), Partition: \([^\]*\).*/ReplicatorException:\1 Topic: \2, Partition: \3/p' @@ -187 +219 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with - <path-to-your-kafka-installation>/bin/query-replicator-failure-message.sh -- replicator-arn $REPLICATOR_ARN --bootstrap-server $TARGET_CLUSTER_SERVER_STRING + <path-to-your-kafka-installation>/bin/query-replicator-failure-message.sh --replicator-arn $REPLICATOR_ARN --bootstrap-server $TARGET_CLUSTER_SERVER_STRING --consumer.config $CONSUMER_CONFIG_FILE @@ -189 +221 @@ Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with -This script outputs all the errors with their exception messages and topic-partition that is experiencing failure. You can use this exception information to mitigate the failures as described in Mitigating MSK Replicator failures. Because the topic contains all the historical failure messages, we recommend that you start investigation using the last message. The following is an example of a failure message. +This script outputs all the errors with their exception messages and affected topic-partitions. You can use this exception information to mitigate the failures as described in Common MSK Replicator failures and their solutions. Because the topic contains all the historical failure messages, start investigation using the last message. The following is an example of a failure message. @@ -196 +228 @@ This script outputs all the errors with their exception messages and topic-parti -### Mitigating MSK Replicator failures +### Common MSK Replicator failures and their solutions @@ -200 +232,6 @@ The following list describes some of the MSK Replicator failures that you might - * **Message size larger than max.request.size** +**Message size larger than max.request.size** + + +###### Cause + +This failure occurs when the MSK Replicator fails to replicate data because the individual message size exceeds 10 MB. By default, MSK Replicator replicates messages up to 10 MB in size. @@ -206 +244,5 @@ The following is an example of this failure message type. -This failure occurs when the MSK Replicator fails to replicate data from the topic because the size of the individual messages being replicated is greater than 10 MB. By default, MSK Replicator replicates messages up to 10 MB in size. To address this, we recommend that you reduce the size of the individual messages in your topic. If you're unable to do so, follow these instructions for [requesting a limit increase](./limits.html#request-msk-quota-increase). +###### Solution + +Reduce the individual message sizes in your topic. If you're unable to do so, follow these instructions for [requesting a limit increase](./limits.html#request-msk-quota-increase). + +**Message size larger than the max message size the server will accept** @@ -208 +250,4 @@ This failure occurs when the MSK Replicator fails to replicate data from the top - * **Message size larger than the max message size the server will accept** + +###### Cause + +This failure occurs when the message size exceeds the target cluster’s maximum message size. @@ -214 +260 @@ The following is an example of this failure message type. -This failure occurs when the size of the messages being replicated exceeds the maximum message size accepted by the target cluster. To mitigate this error, we recommend that you increase the `max.message.bytes` configuration on the target cluster or corresponding target cluster topic to the uncompressed size of the largest message that will be replicated. For information about doing this, see [max.message.bytes](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes). +###### Solution @@ -216 +262,8 @@ This failure occurs when the size of the messages being replicated exceeds the m - * **Timestamp is out of range** +Increase the `max.message.bytes` configuration on the target cluster or corresponding target cluster topic. Set the target cluster’s `max.message.bytes` configuration to match your largest uncompressed message size. For information about doing this, see [max.message.bytes](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes). + +**Timestamp is out of range** + + +###### Cause + +This failure occurs because the individual message timestamp falls outside of the target cluster’s allowed range. @@ -222 +276,3 @@ The following is an example of this failure message type. -This failure occurs because the timestamp of the individual messages being replicated is outside of the range allowed by your target cluster. To mitigate this error, update the target cluster configuration to allow for messages with older timestamps. For information about doing this, see [segment.bytes](https://kafka.apache.org/documentation/#topicconfigs_segment.bytes). +###### Solution + +Update the target cluster’s `message.timestamp.before.max.ms` configuration to allow for messages with older timestamps. For information about doing this, see [message.timestamp.before.max.ms](https://kafka.apache.org/documentation/#topicconfigs_message.timestamp.before.max.ms). @@ -224 +280,6 @@ This failure occurs because the timestamp of the individual messages being repli - * **Record batch too large** +**Record batch too large** + + +###### Cause + +This failure occurs because the record batch size exceeds the segment size set for the topic on the target cluster. MSK Replicator supports a maximum batch size of 1 MB. @@ -228 +288,0 @@ The following is an example of this failure message type. - ReplicatorException: The request included message batch larger than the configured segment size on the server. Topic: test, Partition: 1 @@ -230 +290 @@ The following is an example of this failure message type. -This failure occurs because the batch size of records being produced is greater than the segment size set for the topic on the target cluster. MSK Replicator supports a maximum batch size of 1 MB, and the `segment.bytes` configuration on the target cluster must be at least as large as the batch size (1 MB) for Replicator to proceed without errors. We recommend that you update the `segment.bytes` topic configuration on the target cluster to be greater than or equal to 1048576 (1 MB). For information about doing this, see [segment.bytes](https://kafka.apache.org/documentation/#topicconfigs_segment.bytes). + ReplicatorException: The request included message batch larger than the configured segment size on the server. Topic: test, Partition: 1 @@ -231,0 +292 @@ This failure occurs because the batch size of records being produced is greater +###### Solution @@ -232,0 +294 @@ This failure occurs because the batch size of records being produced is greater +The target cluster’s segment.bytes configuration must be at least as large as the batch size (1 MB) for Replicator to proceed without errors. Update the target cluster’s segment.bytes to be at least 1048576 (1 MB). For information about doing this, see [segment.bytes](https://kafka.apache.org/documentation/#topicconfigs_segment.bytes). @@ -233,0 +296 @@ This failure occurs because the batch size of records being produced is greater +###### Note @@ -235 +298 @@ This failure occurs because the batch size of records being produced is greater -Despite following these mitigation steps, if the ReplicatorFailure metric continues to emit a non-zero value, we recommend that you repeat these steps till the metric no longer emits a non-zero value. +If the ReplicatorFailure metric continues to emit non-zero values after applying these solutions, repeat the troubleshooting process until the metric emits a value of zero.