AWS msk documentation change
Summary
Added new troubleshooting section for MSK Replicator failures including error detection script and mitigation steps for message size limits, timestamp issues, and batch size problems
Security assessment
The changes address operational failures and configuration adjustments for data replication reliability, but do not mention security vulnerabilities, authentication, encryption, or access controls. The added content focuses on troubleshooting message throughput and cluster configuration rather than security aspects.
Diff
diff --git a/msk/latest/developerguide/msk-replicator-troubleshooting.md b/msk/latest/developerguide/msk-replicator-troubleshooting.md index b10e864d5..007fb803d 100644 --- a//msk/latest/developerguide/msk-replicator-troubleshooting.md +++ b//msk/latest/developerguide/msk-replicator-troubleshooting.md @@ -5 +5 @@ -MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in the CREATING stateMSK Replicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterMSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target clusterReplication latency is high or keeps increasing +MSK Replicator state goes from CREATING to FAILEDMSK Replicator appears stuck in the CREATING stateMSK Replicator is not replicating data or replicating only partial dataMessage offsets in the target cluster are different than the source clusterMSK Replicator is not syncing consumer groups offsets or consumer group does not exist on target clusterReplication latency is high or keeps increasingMSK Replicator failure @@ -98,0 +99,138 @@ Throughput and recommended minimum number of partitions Throughput (MB/s) | Mini +## MSK Replicator failed to replicate data + +When data replication issues occur, the ReplicatorFailure metric helps you identify and resolve those problems. When this metric emits a non-zero value, you might experience replication failures related to message size limitations, timestamp range violations, record batch size problems, among others. To restore data replication across your MSK clusters based on the type of error that occurs, perform the following steps. + + 1. Configure a client that is able to connect to the target MSK cluster and has Apache Kafka CLI tools setup. For information about setting up the client and Kafka CLI tool, see [Connect to an Amazon MSK Provisioned cluster](./client-access.html). + + 2. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/home?region=us-east-1#/home/](https://console.aws.amazon.com/msk/home?region=us-east-1#/home/). + +Then, do the following: + + 1. Obtain the ARNs of MSK Replicator and target MSK cluster. + + 2. [Obtain the broker endpoints](./get-bootstrap-console.html) of the target MSK cluster. You'll use these endpoints in the following steps. + + 3. Run the following commands to export the MSK Replicator ARN and broker endpoints you obtained in the previous step. + +Make sure that you replace <`ReplicatorARN`> and <`BootstrapServerString`> with the ARN and broker endpoint that you obtained in the previous step. + + export TARGET_CLUSTER_SERVER_STRING=<BootstrapServerString> + + export REPLICATOR_ARN=<ReplicatorARN> + + 4. In your `<`path-to-your-kafka-installation`>/bin` directory, do the following: + + 1. Save the following script and name it `query-replicator-failure-message.sh`. + + #!/bin/bash + + # Script: Query MSK Replicator Failure Message + # Description: This script queries exceptions from AWS MSK Replicator status topics + # It takes a replicator ARN and bootstrap server as input and searches for replicator exceptions + # in the replicator's status topic, formatting and displaying them in a readable manner + # + # Required Arguments: + # --replicator-arn: The ARN of the AWS MSK Replicator + # --bootstrap-server: The Kafka bootstrap server to connect to + # + # Usage Example: + # ./query-replicator-failure-message.sh --replicator-arn <arn> --bootstrap-server <bootstrap-server> + + print_usage() { + echo "USAGE: $0 --replicator-arn <arn> --bootstrap-server <bootstrap-server>" + echo "--replicator-arn <String: MSK Replicator ARN> REQUIRED: The ARN of AWS MSK Replicator." + echo "--bootstrap-server <String: server to connect to> REQUIRED: The Kafka server to connect to." + } + + if [ $# -eq 2 ]; + then + print_usage + exit 1 + fi + replicator_arn="" + bootstrap_server="" + while [[ $# -gt 0 ]]; do + case "$1" in + --replicator-arn) + replicator_arn="$2" + shift 2 + ;; + --bootstrap-server) + bootstrap_server="$2" + shift 2 + ;; + *) + echo "Unknown option: $1" + print_usage + exit 1 + ;; + esac + done + + replicator_arn_suffix=$(echo "$replicator_arn" | awk -F'/' '{print $NF}') + replicator_name=$(echo "$replicator_arn" | awk -F'/' '{print $(NF-1)}') + echo "Replicator name: $replicator_name" + topics=$(./kafka-topics.sh --command-config client.properties --list --bootstrap-server "$bootstrap_server") + status_topic_name="__amazon_msk_replicator_status_${replicator_name}_${replicator_arn_suffix}" + + # Check if any topic matches the MSK Replicator status topic name + echo "$topics" | grep -F "$status_topic_name" > /dev/null + if [ $? -eq 0 ]; then + echo "Found replicator status topic: '$status_topic_name'" + ./kafka-console-consumer.sh --bootstrap-server $bootstrap_server --consumer.config client.properties --topic "$status_topic_name" --from-beginning --property print.key=true --property key.separator=":" | grep "Exception" | sed -n 's/.*Exception:\(.*\) Topic: \([^,]*\), Partition: \([^\]*\).*/ReplicatorException:\1 Topic: \2, Partition: \3/p' + else + echo "No topic matching the pattern '$status_topic_name' found." + fi + + 2. Run this script to query the MSK Replicator failure messages. + + <path-to-your-kafka-installation>/bin/query-replicator-failure-message.sh -- replicator-arn $REPLICATOR_ARN --bootstrap-server $TARGET_CLUSTER_SERVER_STRING + +This script outputs all the errors with their exception messages and topic-partition that is experiencing failure. You can use this exception information to mitigate the failures as described in Mitigating MSK Replicator failures. Because the topic contains all the historical failure messages, we recommend that you start investigation using the last message. The following is an example of a failure message. + + ReplicatorException: The request included a message larger than the max message size the server will accept. Topic: test, Partition: 1 + + + + +### Mitigating MSK Replicator failures + +The following list describes some of the MSK Replicator failures that you might experience and how to mitigate them. + + * **Message size larger than max.request.size** + +The following is an example of this failure message type. + + ReplicatorException: The message is 20635370 bytes when serialized which is larger than 10485760, which is the value of the max.request.size configuration. Topic: test, Partition: 1 + +This failure occurs when the MSK Replicator fails to replicate data from the topic because the size of the individual messages being replicated is greater than 10 MB. By default, MSK Replicator replicates messages up to 10 MB in size. To address this, we recommend that you reduce the size of the individual messages in your topic. If you're unable to do so, follow these instructions for [requesting a limit increase](./limits.html#request-msk-quota-increase). + + * **Message size larger than the max message size the server will accept** + +The following is an example of this failure message type. + + ReplicatorException: The request included a message larger than the max message size the server will accept. Topic: test, Partition: 1 + +This failure occurs when the size of the messages being replicated exceeds the maximum message size accepted by the target cluster. To mitigate this error, we recommend that you increase the `max.message.bytes` configuration on the target cluster or corresponding target cluster topic to the uncompressed size of the largest message that will be replicated. For information about doing this, see [max.message.bytes](https://kafka.apache.org/documentation/#topicconfigs_max.message.bytes). + + * **Timestamp is out of range** + +The following is an example of this failure message type. + + ReplicatorException: Timestamp 1730137653724 of message with offset 0 is out of range. The timestamp should be within [1730137892239, 1731347492239] Topic: test, Partition: 1 + +This failure occurs because the timestamp of the individual messages being replicated is outside of the range allowed by your target cluster. To mitigate this error, update the target cluster configuration to allow for messages with older timestamps. For information about doing this, see [segment.bytes](https://kafka.apache.org/documentation/#topicconfigs_segment.bytes). + + * **Record batch too large** + +The following is an example of this failure message type. + + ReplicatorException: The request included message batch larger than the configured segment size on the server. Topic: test, Partition: 1 + +This failure occurs because the batch size of records being produced is greater than the segment size set for the topic on the target cluster. MSK Replicator supports a maximum batch size of 1 MB, and the `segment.bytes` configuration on the target cluster must be at least as large as the batch size (1 MB) for Replicator to proceed without errors. We recommend that you update the `segment.bytes` topic configuration on the target cluster to be greater than or equal to 1048576 (1 MB). For information about doing this, see [segment.bytes](https://kafka.apache.org/documentation/#topicconfigs_segment.bytes). + + + + +Despite following these mitigation steps, if the ReplicatorFailure metric continues to emit a non-zero value, we recommend that you repeat these steps till the metric no longer emits a non-zero value. +