AWS lambda documentation change
Summary
Restructured documentation by replacing inline sections with hyperlinks to dedicated pages for MSK event source mapping configuration topics. Removed detailed implementation content that was moved to linked sub-documents.
Security assessment
The changes are organizational/structural improvements to documentation navigation. While authentication methods (SASL/SCRAM, mTLS) and IAM permissions are security-related topics, this diff itself only moves existing security documentation to dedicated pages rather than adding/removing security content. No concrete evidence of addressing a vulnerability.
Diff
diff --git a/lambda/latest/dg/with-msk-configure.md b/lambda/latest/dg/with-msk-configure.md index b31ae6cc7..e1b3834c8 100644 --- a//lambda/latest/dg/with-msk-configure.md +++ b//lambda/latest/dg/with-msk-configure.md @@ -5 +5 @@ -Using an Amazon MSK cluster as an event sourceCreating an event source mapping for an Amazon MSK event sourceConfiguring cluster authentication methodsCustomizable consumer group IDPolling and stream starting positionsEvent poller scaling modesCreating cross-account event source mappingsAll Amazon MSK event source configuration parameters +Amazon MSK cluster as an event source @@ -17 +17 @@ This page assumes that you've already properly configured your MSK cluster and t - * Creating an event source mapping for an Amazon MSK event source + * [Creating a Lambda event source mapping for an Amazon MSK event source](./msk-esm-create.html) @@ -19 +19 @@ This page assumes that you've already properly configured your MSK cluster and t - * Configuring cluster authentication methods + * [Configuring cluster authentication methods in Lambda](./msk-cluster-auth.html) @@ -21 +21 @@ This page assumes that you've already properly configured your MSK cluster and t - * Customizable consumer group ID + * [Customizable consumer group ID in Lambda](./msk-cgid.html) @@ -23 +23 @@ This page assumes that you've already properly configured your MSK cluster and t - * Polling and stream starting positions + * [Polling and stream starting positions in Lambda](./msk-starting-positions.html) @@ -25 +25 @@ This page assumes that you've already properly configured your MSK cluster and t - * Event poller scaling modes + * [Event poller scaling modes in Lambda](./msk-scaling-modes.html) @@ -27 +27 @@ This page assumes that you've already properly configured your MSK cluster and t - * Creating cross-account event source mappings + * [Creating cross-account event source mappings in Lambda](./msk-cross-account.html) @@ -29 +29 @@ This page assumes that you've already properly configured your MSK cluster and t - * All Amazon MSK event source configuration parameters + * [All Amazon MSK event source configuration parameters in Lambda](./msk-esm-parameters.html) @@ -38 +38 @@ When you add your Apache Kafka or Amazon MSK cluster as a trigger for your Lambd -Lambda reads event data from the Kafka topics that you specify as `Topics` in a [CreateEventSourceMapping](https://docs.aws.amazon.com/lambda/latest/api/API_CreateEventSourceMapping.html) request, based on the starting position that you specify. After successful processing, your Kafka topic is committed to your Kafka cluster. +Lambda reads event data from the Kafka topics that you specify as `Topics` in a [CreateEventSourceMapping](https://docs.aws.amazon.com/lambda/latest/api/API_CreateEventSourceMapping.html) request, based on the [starting position](./msk-starting-positions.html) that you specify. After successful processing, your Kafka topic is committed to your Kafka cluster. @@ -48,427 +47,0 @@ While Lambda functions typically have a maximum timeout limit of 15 minutes, eve -## Creating an event source mapping for an Amazon MSK event source - -To create an event source mapping, you can use the Lambda console, the [AWS Command Line Interface (CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), or an [AWS SDK](https://aws.amazon.com/getting-started/tools-sdks/). - -###### Note - -When you create the event source mapping, Lambda creates a [ hyperplane ENI](./configuration-vpc.html#configuration-vpc-enis) in the private subnet that contains your MSK cluster, allowing Lambda to establish a secure connection. This hyperplane ENI allows uses the subnet and security group configuration of your MSK cluster, not your Lambda function. - -The following console steps add an Amazon MSK cluster as a trigger for your Lambda function. Under the hood, this creates an event source mapping resource. - -###### To add an Amazon MSK trigger to your Lambda function (console) - - 1. Open the [Function page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console. - - 2. Choose the name of the Lambda function you want to add an Amazon MSK trigger to. - - 3. Under **Function overview** , choose **Add trigger**. - - 4. Under **Trigger configuration** , choose **MSK**. - - 5. To specify your Kafka cluster details, do the following: - - 1. For **MSK cluster** , select your cluster. - - 2. For **Topic name** , enter the name of the Kafka topic to consume messages from. - - 3. For **Consumer group ID** , enter the ID of a Kafka consumer group to join, if applicable. For more information, see Customizable consumer group ID. - - 6. For **Cluster authentication** , make the necessary configurations. For more information about cluster authentication, see Configuring cluster authentication methods. - - * Toggle on **Use authentication** if you want Lambda to perform authentication with your MSK cluster when establishing a connection. Authentication is recommended. - - * If you use authentication, for **Authentication method** , choose the authentication method to use. - - * If you use authentication, for **Secrets Manager key** , choose the Secrets Manager key that contains the authentication credentials needed to access your cluster. - - 7. Under **Event poller configuration** , make the necessary configurations. - - * Choose **Activate trigger** to enable the trigger immediately after creation. - - * Choose whether you want to **Configure provisioned mode** for your event source mapping. For more information, see Event poller scaling modes. - - * If you configure provisioned mode, enter a value for **Minimum event pollers** , a value for **Maximum event pollers** , or both values. - - * For **Starting position** , choose how you want Lambda to start reading from your stream. For more information, see Polling and stream starting positions. - - 8. Under **Batching** , make the necessary configurations. For more information about batching, see [Batching behavior](./invocation-eventsourcemapping.html#invocation-eventsourcemapping-batching). - - 1. For **Batch size** , enter the maximum number of messages to receive in a single batch. - - 2. For **Batch window** , enter the maximum number of seconds that Lambda spends gathering records before invoking the function. - - 9. Under **Filtering** , make the necessary configurations. For more information about filtering, see [Using event filtering with an Amazon MSK event source](./with-msk-filtering.html). - - * For **Filter criteria** , add filter criteria definitions to determine whether or not to process an event. - - 10. Under **Failure handling** , make the necessary configurations. For more information about failure handling, see [Capturing discarded batches for an Amazon MSK event source](./with-msk-on-failure.html). - - * For **On-failure destination** , specify the ARN of your on-failure destination. - - 11. For **Tags** , enter the tags to associate with this event source mapping. - - 12. To create the trigger, choose **Add**. - - - - -You can also create the event source mapping using the AWS CLI with the [ create-event-source-mapping](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/create-event-source-mapping.html) command. The following example creates an event source mapping to map the Lambda function `my-msk-function` to the `AWSKafkaTopic` topic, starting from the `LATEST` message. This command also uses the [SourceAccessConfiguration](https://docs.aws.amazon.com/lambda/latest/api/API_SourceAccessConfiguration.html) object to instruct Lambda to use SASL/SCRAM authentication when connecting to the cluster. - - - aws lambda create-event-source-mapping \ - --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \ - --topics AWSKafkaTopic \ - --starting-position LATEST \ - --function-name my-kafka-function - --source-access-configurations '[{"Type": "SASL_SCRAM_512_AUTH","URI": "arn:aws:secretsmanager:us-east-1:111122223333:secret:my-secret"}]' - -If the cluster uses mTLS authentication, include a [SourceAccessConfiguration](https://docs.aws.amazon.com/lambda/latest/api/API_SourceAccessConfiguration.html) object that specifies `CLIENT_CERTIFICATE_TLS_AUTH` and a Secrets Manager key ARN. This is shown in the following command: - - - aws lambda create-event-source-mapping \ - --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \ - --topics AWSKafkaTopic \ - --starting-position LATEST \ - --function-name my-kafka-function - --source-access-configurations '[{"Type": "CLIENT_CERTIFICATE_TLS_AUTH","URI": "arn:aws:secretsmanager:us-east-1:111122223333:secret:my-secret"}]' - -When the cluster uses IAM authentication, you don’t need a [ SourceAccessConfiguration](https://docs.aws.amazon.com/lambda/latest/api/API_SourceAccessConfiguration.html) object. This is shown in the following command: - - - aws lambda create-event-source-mapping \ - --event-source-arn arn:aws:kafka:us-east-1:111122223333:cluster/my-cluster/fc2f5bdf-fd1b-45ad-85dd-15b4a5a6247e-2 \ - --topics AWSKafkaTopic \ - --starting-position LATEST \ - --function-name my-kafka-function - -## Configuring cluster authentication methods - -Lambda needs permission to access your Amazon MSK cluster, retrieve records, and perform other tasks. Amazon MSK supports several ways to authenticate with your MSK cluster. - -###### Cluster authentication methods - - * Unauthenticated access - - * SASL/SCRAM authentication - - * Mutual TLS authentication - - * IAM authentication - - * How Lambda chooses a bootstrap broker - - - - -### Unauthenticated access - -If no clients access the cluster over the internet, you can use unauthenticated access. - -### SASL/SCRAM authentication - -Lambda supports [ Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial.html) authentication, with the SHA-512 hash function and Transport Layer Security (TLS) encryption. For Lambda to connect to the cluster, store the authentication credentials (username and password) in a Secrets Manager secret, and reference this secret when configuring your event source mapping. - -For more information about using Secrets Manager, see [Sign-in credentials authentication with Secrets Manager](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html) in the _Amazon Managed Streaming for Apache Kafka Developer Guide_. - -###### Note - -Amazon MSK doesn’t support SASL/PLAIN authentication. - -### Mutual TLS authentication - -Mutual TLS (mTLS) provides two-way authentication between the client and the server. The client sends a certificate to the server for the server to verify the client. The server also sends a certificate to the client for the client to verify the server. - -For Amazon MSK integrations with Lambda, your MSK cluster acts as the server, and Lambda acts as the client. - - * For Lambda to verify your MSK cluster, you configure a client certificate as a secret in Secrets Manager, and reference this certificate in your event source mapping configuration. The client certificate must be signed by a certificate authority (CA) in the server’s trust store. - - * The MSK cluster also sends a server certificate to Lambda. The server certificate must be signed by a certificate authority (CA) in the AWS trust store. - - - - -Amazon MSK doesn’t support self-signed server certificates. All brokers in Amazon MSK use [public certificates](https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.html) signed by [Amazon Trust Services CAs](https://www.amazontrust.com/repository/), which Lambda trusts by default. - -The CLIENT_CERTIFICATE_TLS_AUTH secret requires a certificate field and a private key field. For an encrypted private key, the secret requires a private key password. Both the certificate and private key must be in PEM format. - -###### Note - -Lambda supports the [PBES1](https://datatracker.ietf.org/doc/html/rfc2898/#section-6.1) (but not PBES2) private key encryption algorithms. - -The certificate field must contain a list of certificates, beginning with the client certificate, followed by any intermediate certificates, and ending with the root certificate. Each certificate must start on a new line with the following structure: - - - -----BEGIN CERTIFICATE----- - <certificate contents> - -----END CERTIFICATE----- - -Secrets Manager supports secrets up to 65,536 bytes, which is enough space for long certificate chains. - -The private key must be in [PKCS #8](https://datatracker.ietf.org/doc/html/rfc5208) format, with the following structure: - - - -----BEGIN PRIVATE KEY----- - <private key contents> - -----END PRIVATE KEY----- - -For an encrypted private key, use the following structure: -