AWS msk documentation change
Summary
Added troubleshooting guidance for client.properties file path issues
Security assessment
Addresses operational errors related to file paths but does not involve security mechanisms or vulnerabilities.
Diff
diff --git a/msk/latest/developerguide/create-topic.md b/msk/latest/developerguide/create-topic.md index 44291ea5a..76956eaa1 100644 --- a//msk/latest/developerguide/create-topic.md +++ b//msk/latest/developerguide/create-topic.md @@ -46 +46 @@ To find the version of your MSK cluster -For example, if you want to use Amazon MSK with Apache Kafka version 3.5.1, run the following command. +For example, if your MSK cluster uses Apache Kafka version 3.5.1, run the following command. @@ -52 +52,9 @@ For example, if you want to use Amazon MSK with Apache Kafka version 3.5.1, run -If you want to use a mirror site other than the one used in this command, you can choose a different one on the [Apache](https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz) website. +The following list presents some alternative Kafka download information that you can use, if you encounter any issues. + + * If you want to use a mirror site other than the one used in this command, you can choose a different one on the [Apache](https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz) website. + + * If you encounter connectivity issues, try using the Apache mirror selector, as shown in the following command. + + wget https://www.apache.org/dyn/closer.cgi?path=/kafka/{YOUR MSK VERSION}/kafka_2.13-{YOUR MSK VERSION}.tgz + + * Download an appropriate version directly from the Apache Kafka website: [Apache Kafka download](https://kafka.apache.org/downloads). @@ -63,0 +72,12 @@ Using this command, you can also [download the latest version](https://github.co +###### Note + +Before running any Kafka commands that interact with your MSK cluster, you must add the AWS MSK IAM JAR file to your Java classpath. This JAR enables IAM authentication between your client and the MSK cluster. Set the `CLASSPATH` environment variable, as shown in the following example. + + export CLASSPATH=<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar + +For example, suppose that you downloaded Kafka to the default location. + + export CLASSPATH=/home/ec2-user/kafka_2.13-3.5.1/libs/aws-msk-iam-auth-2.3.0-all.jar + +This sets the `CLASSPATH` for your entire session, making the JAR available to all subsequent Kafka commands. + @@ -71 +91,5 @@ Using this command, you can also [download the latest version](https://github.co - 9. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). + 9. (Optional) If you need to adjust the Java heap size for Kafka tools, you can set the `KAFKA_HEAP_OPTS` environment variable before running Kafka commands. This is useful if you encounter memory-related issues or are working with a large number of topics or partitions. + +The following example sets both the maximum and initial heap size to 512 megabytes. Adjust these values according to your specific requirements and available system resources. + + export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M" @@ -73 +97 @@ Using this command, you can also [download the latest version](https://github.co - 10. Wait for the status of your cluster to become **Active**. This might take several minutes. After the status becomes **Active** , choose the cluster name. This takes you to a page containing the cluster summary. + 10. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). @@ -75 +99 @@ Using this command, you can also [download the latest version](https://github.co - 11. Choose **View client information**. + 11. Wait for the status of your cluster to become **Active**. This might take several minutes. After the status becomes **Active** , choose the cluster name. This takes you to a page containing the cluster summary. @@ -77 +101,3 @@ Using this command, you can also [download the latest version](https://github.co - 12. Copy the connection string for the private endpoint. + 12. Choose **View client information**. + + 13. Copy the connection string for the private endpoint. @@ -81 +107 @@ You will get three endpoints for each of the brokers. You only need one broker e - 13. Run the following command, replacing `BootstrapServerString` with one of the broker endpoints that you obtained in the previous step. + 14. Run the following command, replacing `BootstrapServerString` with one of the broker endpoints that you obtained in the previous step. @@ -85 +111 @@ You will get three endpoints for each of the brokers. You only need one broker e -###### Note +If you get a `NoSuchFileException` for the `client.properties` file, make sure that this file exists in the current working directory within the Kafka bin directory. Alternatively, provide a full path to this file's location in the Kafka command, as shown in the following example. @@ -87 +113 @@ You will get three endpoints for each of the brokers. You only need one broker e -Before running this command, you must add the AWS MSK IAM JAR file to your classpath to avoid the `Class not found` error. You can do this by setting the `CLASSPATH` environment variable, as shown in the following command. + <path-to-your-kafka-installation>/bin/kafka-topics.sh --create --bootstrap-server BootstrapServerString --command-config <path-to-client-properties>/client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic @@ -89 +115,3 @@ Before running this command, you must add the AWS MSK IAM JAR file to your class - export CLASSPATH=<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar +###### Note + +If you prefer not to set the `CLASSPATH` environment variable for your entire session, you can alternatively prefix each Kafka command with the `CLASSPATH` variable. This approach applies the classpath only to that specific command. @@ -91 +119 @@ Before running this command, you must add the AWS MSK IAM JAR file to your class -Alternatively, you can also modify your command to explicitly add the JAR to the `classpath`, as shown in the following example. +For example, if you've installed Kafka in `/home/ec2-user/kafka_2.13-3.5.1` and your broker endpoint is `myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098:`, prefix the Kafka command with the `CLASSPATH` variable, as shown in the following example. @@ -93,3 +121,3 @@ Alternatively, you can also modify your command to explicitly add the JAR to the - classpath=<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar \ - <path-to-your-kafka-installation>/bin/kafka-topics.sh --create \ - --bootstrap-server BootstrapServerString \ + CLASSPATH=/home/ec2-user/kafka_2.13-3.5.1/libs/aws-msk-iam-auth-2.3.0-all.jar \ + /home/ec2-user/kafka_2.13-3.5.1/bin/kafka-topics.sh --create \ + --bootstrap-server myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098 \ @@ -101,5 +128,0 @@ Alternatively, you can also modify your command to explicitly add the JAR to the -For example, if Apache Kafka is installed in `/home/ec2-user/kafka_2.13-2.8.1`, and your broker endpoint is `myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098`, you'd run the following command. - - export CLASSPATH=/home/ec2-user/kafka_2.13-2.8.1/libs/aws-msk-iam-auth-2.3.0-all.jar - /home/ec2-user/kafka_2.13-2.8.1/bin/kafka-topics.sh --create --bootstrap-server myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098 --command-config client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic -