AWS msk documentation change
Summary
Updated documentation for creating Kafka topics with improved structure, environment variable usage, and explicit IAM authentication setup steps
Security assessment
The changes explicitly document IAM authentication setup through the Amazon MSK IAM JAR file and classpath configuration. While this relates to security best practices, there is no indication of addressing a specific vulnerability. The updates improve clarity around secure access configuration but don't reference any CVE or security incident.
Diff
diff --git a/msk/latest/developerguide/create-topic.md b/msk/latest/developerguide/create-topic.md index 76956eaa1..b1fd70cea 100644 --- a//msk/latest/developerguide/create-topic.md +++ b//msk/latest/developerguide/create-topic.md @@ -4,0 +5,2 @@ +Find your cluster versionCreate a topic + @@ -13 +15,6 @@ Apache Kafka version numbers used in this tutorial are examples only. We recomme -###### +###### Topics + + * Determining your MSK cluster version + + * Creating a topic on the client machine + @@ -15 +22,3 @@ Apache Kafka version numbers used in this tutorial are examples only. We recomme -To find the version of your MSK cluster + + +## Determining your MSK cluster version @@ -30 +39,3 @@ To find the version of your MSK cluster -###### To create a topic on the client machine +## Creating a topic on the client machine + + 1. **Connect to your client machine.** @@ -34 +45 @@ To find the version of your MSK cluster - 2. In the navigation pane, choose **Instances**. Then select the check box beside the name of the client machine that you created in [Step 3: Create a client machine](./create-client-machine.html). + 2. In the navigation pane, choose **Instances**. Then, select the check box beside the name of the client machine that you created in [Step 3: Create a client machine](./create-client-machine.html). @@ -38 +49,3 @@ To find the version of your MSK cluster - 4. Install Java on the client machine by running the following command: + 2. **Install Java and set up the Kafka version environment variable.** + + 1. Install Java on the client machine by running the following command. @@ -42 +55,11 @@ To find the version of your MSK cluster - 5. Run the following command to download Apache Kafka. + 2. Store the Kafka version of your MSK cluster in the environment variable, `KAFKA_VERSION`, as shown in the following command. You'll need this information throughout the setup. + + export KAFKA_VERSION={KAFKA VERSION} + +For example, if you're using version 3.6.0, use the following command. + + export KAFKA_VERSION=3.6.0 + + 3. **Download and extract Apache Kafka.** + + 1. Run the following command to download Apache Kafka. @@ -44 +67 @@ To find the version of your MSK cluster - wget https://archive.apache.org/dist/kafka/{YOUR MSK VERSION}/kafka_2.13-{YOUR MSK VERSION}.tgz + wget https://archive.apache.org/dist/kafka/$KAFKA_VERSION/kafka_2.13-$KAFKA_VERSION.tgz @@ -46 +69 @@ To find the version of your MSK cluster -For example, if your MSK cluster uses Apache Kafka version 3.5.1, run the following command. +For example, if your MSK cluster uses Apache Kafka version 3.6.0, run the following command. @@ -48 +71 @@ For example, if your MSK cluster uses Apache Kafka version 3.5.1, run the follow - wget https://archive.apache.org/dist/kafka/3.5.1/kafka_2.13-3.5.1.tgz + wget https://archive.apache.org/dist/kafka/3.6.0/kafka_2.13-3.6.0.tgz @@ -54 +77 @@ The following list presents some alternative Kafka download information that you - * If you want to use a mirror site other than the one used in this command, you can choose a different one on the [Apache](https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.1/kafka_2.11-1.1.1.tgz) website. + * If you encounter connectivity issues or want to use a mirror site, try using the Apache mirror selector, as shown in the following command. @@ -56 +79 @@ The following list presents some alternative Kafka download information that you - * If you encounter connectivity issues, try using the Apache mirror selector, as shown in the following command. + wget https://www.apache.org/dyn/closer.cgi?path=/kafka/$KAFKA_VERSION/kafka_2.13-$KAFKA_VERSION.tgz @@ -58 +81 @@ The following list presents some alternative Kafka download information that you - wget https://www.apache.org/dyn/closer.cgi?path=/kafka/{YOUR MSK VERSION}/kafka_2.13-{YOUR MSK VERSION}.tgz + * Download an appropriate version directly from the [Apache Kafka website](https://kafka.apache.org/downloads). @@ -60 +83 @@ The following list presents some alternative Kafka download information that you - * Download an appropriate version directly from the Apache Kafka website: [Apache Kafka download](https://kafka.apache.org/downloads). + 2. Run the following command in the directory where you downloaded the TAR file in the previous step. @@ -62 +85 @@ The following list presents some alternative Kafka download information that you - 6. Run the following command in the directory where you downloaded the TAR file in the previous step. + tar -xzf kafka_2.13-$KAFKA_VERSION.tgz @@ -64 +87 @@ The following list presents some alternative Kafka download information that you - tar -xzf kafka_2.13-{YOUR MSK VERSION}.tgz + 3. Store the full path to the newly created directory inside the `KAFKA_ROOT` environment variable. @@ -66 +89 @@ The following list presents some alternative Kafka download information that you - 7. Go to the `kafka_2.13-{YOUR MSK VERSION}/libs` directory, then run the following command to download the Amazon MSK IAM JAR file. The Amazon MSK IAM JAR makes it possible for the client machine to access the cluster. + export KAFKA_ROOT=$(pwd)/kafka_2.13-$KAFKA_VERSION @@ -68 +91 @@ The following list presents some alternative Kafka download information that you - wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.3.0/aws-msk-iam-auth-2.3.0-all.jar + 4. **Set up authentication for your MSK cluster.** @@ -70 +93 @@ The following list presents some alternative Kafka download information that you -Using this command, you can also [download the latest version](https://github.com/aws/aws-msk-iam-auth/releases) of `aws-msk-iam-auth-*-all.jar`. + 1. Download the latest version of the Amazon MSK IAM JAR file inside the `$KAFKA_ROOT/libs` directory. Use the following command to download the file, and replace `{LATEST VERSION}` with the actual version number. @@ -72 +95 @@ Using this command, you can also [download the latest version](https://github.co -###### Note + cd $KAFKA_ROOT/libs wget https://github.com/aws/aws-msk-iam-auth/releases/latest/download/aws-msk-iam-auth-{LATEST VERSION}-all.jar @@ -74 +97 @@ Using this command, you can also [download the latest version](https://github.co -Before running any Kafka commands that interact with your MSK cluster, you must add the AWS MSK IAM JAR file to your Java classpath. This JAR enables IAM authentication between your client and the MSK cluster. Set the `CLASSPATH` environment variable, as shown in the following example. +The Amazon MSK IAM JAR file allows your client machine to access the MSK cluster using IAM authentication. @@ -76 +99 @@ Before running any Kafka commands that interact with your MSK cluster, you must - export CLASSPATH=<path-to-your-kafka-installation>/libs/aws-msk-iam-auth-2.3.0-all.jar +###### Note @@ -78 +101 @@ Before running any Kafka commands that interact with your MSK cluster, you must -For example, suppose that you downloaded Kafka to the default location. +Before running any Kafka commands that interact with your MSK cluster, you might need to add the Amazon MSK IAM JAR file to your Java classpath. Set the `CLASSPATH` environment variable, as shown in the following example. @@ -80 +103 @@ For example, suppose that you downloaded Kafka to the default location. - export CLASSPATH=/home/ec2-user/kafka_2.13-3.5.1/libs/aws-msk-iam-auth-2.3.0-all.jar + export CLASSPATH=$KAFKA_ROOT/libs/aws-msk-iam-auth-{LATEST VERSION}-all.jar @@ -84 +107,5 @@ This sets the `CLASSPATH` for your entire session, making the JAR available to a - 8. Go to the `kafka_2.13-{YOUR MSK VERSION}/config` directory. Copy the following property settings and paste them into a new file. Name the file `client.properties` and save it. + 2. Go to the `$KAFKA_ROOT/config` directory to create the client configuration file. + + cd $KAFKA_ROOT/config + + 3. Copy the following property settings and paste them into a new file. Save the file as `client.properties`. @@ -91 +118 @@ This sets the `CLASSPATH` for your entire session, making the JAR available to a - 9. (Optional) If you need to adjust the Java heap size for Kafka tools, you can set the `KAFKA_HEAP_OPTS` environment variable before running Kafka commands. This is useful if you encounter memory-related issues or are working with a large number of topics or partitions. + 5. (Optional) If you encounter any memory-related issues or are working with a large number of topics or partitions, you can adjust the Java heap size for Kafka tools. To do this, set the `KAFKA_HEAP_OPTS` environment variable before running Kafka commands. @@ -97 +124,3 @@ The following example sets both the maximum and initial heap size to 512 megabyt - 10. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). + 6. **Get your cluster connection information.** + + 1. Open the Amazon MSK console at [https://console.aws.amazon.com/msk/](https://console.aws.amazon.com/msk/). @@ -99 +128 @@ The following example sets both the maximum and initial heap size to 512 megabyt - 11. Wait for the status of your cluster to become **Active**. This might take several minutes. After the status becomes **Active** , choose the cluster name. This takes you to a page containing the cluster summary. + 2. Wait for the status of your cluster to become **Active**. This might take several minutes. After the status becomes **Active** , choose the cluster name. This takes you to a page containing the cluster summary. @@ -101 +130 @@ The following example sets both the maximum and initial heap size to 512 megabyt - 12. Choose **View client information**. + 3. Choose **View client information**. @@ -103 +132 @@ The following example sets both the maximum and initial heap size to 512 megabyt - 13. Copy the connection string for the private endpoint. + 4. Copy the connection string for the private endpoint. @@ -105 +134 @@ The following example sets both the maximum and initial heap size to 512 megabyt -You will get three endpoints for each of the brokers. You only need one broker endpoint for the following step. +You'll get three endpoints for each of the brokers. Store one of these connection strings in the environment variable `BOOTSTRAP_SERVER`, as shown in the following command. Replace `<bootstrap-server-string>` with the actual value of the connection string. @@ -107 +136 @@ You will get three endpoints for each of the brokers. You only need one broker e - 14. Run the following command, replacing `BootstrapServerString` with one of the broker endpoints that you obtained in the previous step. + export BOOTSTRAP_SERVER=<bootstrap-server-string> @@ -109 +138 @@ You will get three endpoints for each of the brokers. You only need one broker e - <path-to-your-kafka-installation>/bin/kafka-topics.sh --create --bootstrap-server BootstrapServerString --command-config client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic + 7. Run the following command to create the topic. @@ -111 +140 @@ You will get three endpoints for each of the brokers. You only need one broker e -If you get a `NoSuchFileException` for the `client.properties` file, make sure that this file exists in the current working directory within the Kafka bin directory. Alternatively, provide a full path to this file's location in the Kafka command, as shown in the following example. + $KAFKA_ROOT/bin/kafka-topics.sh --create --bootstrap-server $BOOTSTRAP_SERVER --command-config $KAFKA_ROOT/config/client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic @@ -113 +142 @@ If you get a `NoSuchFileException` for the `client.properties` file, make sure t - <path-to-your-kafka-installation>/bin/kafka-topics.sh --create --bootstrap-server BootstrapServerString --command-config <path-to-client-properties>/client.properties --replication-factor 3 --partitions 1 --topic MSKTutorialTopic +If you get a `NoSuchFileException` for the `client.properties` file, make sure that this file exists in the current working directory within the Kafka bin directory. @@ -119,6 +148,4 @@ If you prefer not to set the `CLASSPATH` environment variable for your entire se -For example, if you've installed Kafka in `/home/ec2-user/kafka_2.13-3.5.1` and your broker endpoint is `myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098:`, prefix the Kafka command with the `CLASSPATH` variable, as shown in the following example. - - CLASSPATH=/home/ec2-user/kafka_2.13-3.5.1/libs/aws-msk-iam-auth-2.3.0-all.jar \ - /home/ec2-user/kafka_2.13-3.5.1/bin/kafka-topics.sh --create \ - --bootstrap-server myBrokerEndpoint-1.myCluster.abc123.kafka.us-east-1.amazonaws.com:9098 \ - --command-config client.properties \ + CLASSPATH=$KAFKA_ROOT/libs/aws-msk-iam-auth-{LATEST VERSION}-all.jar \ + $KAFKA_ROOT/bin/kafka-topics.sh --create \ + --bootstrap-server $BOOTSTRAP_SERVER \ + --command-config $KAFKA_ROOT/config/client.properties \ @@ -129 +156,7 @@ For example, if you've installed Kafka in `/home/ec2-user/kafka_2.13-3.5.1` and -If the command succeeds, you see the following message: `Created topic MSKTutorialTopic.` + 8. **(Optional) Verify that the topic was created successfully.** + + 1. If the command succeeds, you should see the following message: `Created topic MSKTutorialTopic.` + + 2. List all topics to confirm your topic exists. + + $KAFKA_ROOT/bin/kafka-topics.sh --list --bootstrap-server $BOOTSTRAP_SERVER --command-config $KAFKA_ROOT/config/client.properties @@ -132,0 +166,4 @@ If the command is unsuccessful or you run into an error, see [Troubleshoot your + 9. (Optional) If you want to keep your environment variables for the next steps in this tutorial, skip this step. Otherwise, you can unset the variables, as shown in the following example. + + unset KAFKA_VERSION KAFKA_ROOT BOOTSTRAP_SERVER CLASSPATH KAFKA_HEAP_OPTS +