AWS lambda documentation change
Summary
Complete restructuring of Amazon MSK event source mapping documentation. Added sections for creating mappings, authentication methods (SASL/SCRAM, mTLS, IAM), consumer group IDs, poller scaling modes, cross-account configurations, and full parameter references. Removed older authentication error troubleshooting and network security details.
Security assessment
The changes extensively document security-related configurations like mTLS certificate requirements, IAM authentication policies, SASL/SCRAM credential management, and cross-account security considerations. However, there's no evidence of addressing a specific disclosed vulnerability or security incident - rather, this is routine documentation of existing security features.
Diff
diff --git a/lambda/latest/dg/with-msk-configure.md b/lambda/latest/dg/with-msk-configure.md index 0603d5acc..01594697a 100644 --- a//lambda/latest/dg/with-msk-configure.md +++ b//lambda/latest/dg/with-msk-configure.md @@ -5 +5 @@ -MSK cluster authenticationManaging API access and permissionsAuthentication and authorization errorsConfigure network security +Using an Amazon MSK cluster as an event sourceCreating an event source mapping for an Amazon MSK event sourceConfiguring cluster authentication methodsCustomizable consumer group IDPolling and stream starting positionsEvent poller scaling modesCreating cross-account event source mappingsAll Amazon MSK event source configuration parameters @@ -9 +9 @@ MSK cluster authenticationManaging API access and permissionsAuthentication and -Before you create an event source mapping for your Amazon MSK cluster, you need to ensure that your cluster and the VPC it resides in are correctly configured. You also need to make sure that your Lambda function's [execution role](./lambda-intro-execution-role.html) has the necessary IAM permissions. +To use an Amazon MSK cluster as an event source for your Lambda function, you create an [event source mapping](./invocation-eventsourcemapping.html) that connects the two resources. This page describes how to create an event source mapping for Amazon MSK. @@ -11 +11 @@ Before you create an event source mapping for your Amazon MSK cluster, you need -Follow the instructions in the following sections to configure your Amazon MSK cluster, VPC, and Lambda function. To learn how to create the event source mapping, see [Adding Amazon MSK as an event source](./with-msk-process.html#services-msk-topic-add). +This page assumes that you've already properly configured your MSK cluster and the [Amazon Virtual Private Cloud (VPC)](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) it resides in. If you need to set up your cluster or VPC, see [Configuring your Amazon MSK cluster and Amazon VPC network for Lambda](./with-msk-cluster-network.html). @@ -15 +15 @@ Follow the instructions in the following sections to configure your Amazon MSK c - * MSK cluster authentication + * Using an Amazon MSK cluster as an event source @@ -17 +17 @@ Follow the instructions in the following sections to configure your Amazon MSK c - * Managing API access and permissions + * Creating an event source mapping for an Amazon MSK event source @@ -19 +19 @@ Follow the instructions in the following sections to configure your Amazon MSK c - * Authentication and authorization errors + * Configuring cluster authentication methods @@ -21 +21 @@ Follow the instructions in the following sections to configure your Amazon MSK c - * Configure network security + * Customizable consumer group ID @@ -22,0 +23 @@ Follow the instructions in the following sections to configure your Amazon MSK c + * Polling and stream starting positions @@ -23,0 +25 @@ Follow the instructions in the following sections to configure your Amazon MSK c + * Event poller scaling modes @@ -24,0 +27 @@ Follow the instructions in the following sections to configure your Amazon MSK c + * Creating cross-account event source mappings @@ -26 +29 @@ Follow the instructions in the following sections to configure your Amazon MSK c -## MSK cluster authentication + * All Amazon MSK event source configuration parameters @@ -28 +30,0 @@ Follow the instructions in the following sections to configure your Amazon MSK c -Lambda needs permission to access the Amazon MSK cluster, retrieve records, and perform other tasks. Amazon MSK supports several options for controlling client access to the MSK cluster. @@ -30 +31,0 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and -###### Cluster access options @@ -32 +32,0 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * Unauthenticated access @@ -34 +34 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * SASL/SCRAM authentication +## Using an Amazon MSK cluster as an event source @@ -36 +36 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * IAM role-based authentication +When you add your Apache Kafka or Amazon MSK cluster as a trigger for your Lambda function, the cluster is used as an [event source](./invocation-eventsourcemapping.html). @@ -38 +38 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * Mutual TLS authentication +Lambda reads event data from the Kafka topics that you specify as `Topics` in a [CreateEventSourceMapping](https://docs.aws.amazon.com/lambda/latest/api/API_CreateEventSourceMapping.html) request, based on the starting position that you specify. After successful processing, your Kafka topic is committed to your Kafka cluster. @@ -40 +40 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * Configuring the mTLS secret +Lambda reads messages sequentially for each Kafka topic partition. A single Lambda payload can contain messages from multiple partitions. When more records are available, Lambda continues processing records in batches, based on the BatchSize value that you specify in a [CreateEventSourceMapping](https://docs.aws.amazon.com/lambda/latest/api/API_CreateEventSourceMapping.html) request, until your function catches up with the topic. @@ -42 +42 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and - * How Lambda chooses a bootstrap broker +After Lambda processes each batch, it commits the offsets of the messages in that batch. If your function returns an error for any of the messages in a batch, Lambda retries the whole batch of messages until processing succeeds or the messages expire. You can send records that fail all retry attempts to an on-failure destination for later processing. @@ -43,0 +44 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and +###### Note @@ -44,0 +46 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and +While Lambda functions typically have a maximum timeout limit of 15 minutes, event source mappings for Amazon MSK, self-managed Apache Kafka, Amazon DocumentDB, and Amazon MQ for ActiveMQ and RabbitMQ only support functions with maximum timeout limits of 14 minutes. @@ -45,0 +48 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and +## Creating an event source mapping for an Amazon MSK event source @@ -47 +50 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and -### Unauthenticated access +To create an event source mapping, you can use the Lambda console, the [AWS Command Line Interface (CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), or an [AWS SDK](https://aws.amazon.com/getting-started/tools-sdks/). @@ -49 +52 @@ Lambda needs permission to access the Amazon MSK cluster, retrieve records, and -If no clients access the cluster over the internet, you can use unauthenticated access. +###### Note @@ -51 +54 @@ If no clients access the cluster over the internet, you can use unauthenticated -### SASL/SCRAM authentication +When you create the event source mapping, Lambda creates a [ hyperplane ENI](./configuration-vpc.html#configuration-vpc-enis) in the private subnet that contains your MSK cluster, allowing Lambda to establish a secure connection. This hyperplane ENI allows uses the subnet and security group configuration of your MSK cluster, not your Lambda function. @@ -53 +56 @@ If no clients access the cluster over the internet, you can use unauthenticated -Amazon MSK supports Simple Authentication and Security Layer/Salted Challenge Response Authentication Mechanism (SASL/SCRAM) authentication with Transport Layer Security (TLS) encryption. For Lambda to connect to the cluster, you store the authentication credentials (user name and password) in an AWS Secrets Manager secret. +The following console steps add an Amazon MSK cluster as a trigger for your Lambda function. Under the hood, this creates an event source mapping resource. @@ -55 +58 @@ Amazon MSK supports Simple Authentication and Security Layer/Salted Challenge Re -For more information about using Secrets Manager, see [User name and password authentication with AWS Secrets Manager](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html) in the _Amazon Managed Streaming for Apache Kafka Developer Guide_. +###### To add an Amazon MSK trigger to your Lambda function (console) @@ -57 +60 @@ For more information about using Secrets Manager, see [User name and password au -Amazon MSK doesn't support SASL/PLAIN authentication. + 1. Open the [Function page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console. @@ -59 +62 @@ Amazon MSK doesn't support SASL/PLAIN authentication. -### IAM role-based authentication + 2. Choose the name of the Lambda function you want to add an Amazon MSK trigger to. @@ -61 +64 @@ Amazon MSK doesn't support SASL/PLAIN authentication. -You can use IAM to authenticate the identity of clients that connect to the MSK cluster. If IAM auth is active on your MSK cluster, and you don't provide a secret for auth, Lambda automatically defaults to using IAM auth. To create and deploy user or role-based policies, use the IAM console or API. For more information, see [IAM access control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html) in the _Amazon Managed Streaming for Apache Kafka Developer Guide_. + 3. Under **Function overview** , choose **Add trigger**. @@ -63 +66 @@ You can use IAM to authenticate the identity of clients that connect to the MSK -To allow Lambda to connect to the MSK cluster, read records, and perform other required actions, add the following permissions to your function's [execution role](./lambda-intro-execution-role.html). + 4. Under **Trigger configuration** , choose **MSK**. @@ -64,0 +68 @@ To allow Lambda to connect to the MSK cluster, read records, and perform other r + 5. To specify your Kafka cluster details, do the following: @@ -66,21 +70 @@ To allow Lambda to connect to the MSK cluster, read records, and perform other r - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "kafka-cluster:Connect", - "kafka-cluster:DescribeGroup", - "kafka-cluster:AlterGroup", - "kafka-cluster:DescribeTopic", - "kafka-cluster:ReadData", - "kafka-cluster:DescribeClusterDynamicConfiguration" - ], - "Resource": [ - "arn:aws:kafka:region:account-id:cluster/cluster-name/cluster-uuid", - "arn:aws:kafka:region:account-id:topic/cluster-name/cluster-uuid/topic-name", - "arn:aws:kafka:region:account-id:group/cluster-name/cluster-uuid/consumer-group-id" - ] - } - ] - } + 1. For **MSK cluster** , select your cluster. @@ -88 +72 @@ To allow Lambda to connect to the MSK cluster, read records, and perform other r -You can scope these permissions to a specific cluster, topic, and group. For more information, see the [Amazon MSK Kafka actions](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#kafka-actions) in the _Amazon Managed Streaming for Apache Kafka Developer Guide_. + 2. For **Topic name** , enter the name of the Kafka topic to consume messages from. @@ -90 +74 @@ You can scope these permissions to a specific cluster, topic, and group. For mor -### Mutual TLS authentication + 3. For **Consumer group ID** , enter the ID of a Kafka consumer group to join, if applicable. For more information, see Customizable consumer group ID. @@ -92 +76 @@ You can scope these permissions to a specific cluster, topic, and group. For mor -Mutual TLS (mTLS) provides two-way authentication between the client and server. The client sends a certificate to the server for the server to verify the client, and the server sends a certificate to the client for the client to verify the server. + 6. For **Cluster authentication** , make the necessary configurations. For more information about cluster authentication, see Configuring cluster authentication methods. @@ -94 +78 @@ Mutual TLS (mTLS) provides two-way authentication between the client and server. -For Amazon MSK, Lambda acts as the client. You configure a client certificate (as a secret in Secrets Manager) to authenticate Lambda with the brokers in your MSK cluster. The client certificate must be signed by a CA in the server's trust store. The MSK cluster sends a server certificate to Lambda to authenticate the brokers with Lambda. The server certificate must be signed by a certificate authority (CA) that's in the AWS trust store. + * Toggle on **Use authentication** if you want Lambda to perform authentication with your MSK cluster when establishing a connection. Authentication is recommended. @@ -96 +80 @@ For Amazon MSK, Lambda acts as the client. You configure a client certificate (a -For instructions on how to generate a client certificate, see [ Introducing mutual TLS authentication for Amazon MSK as an event source](https://aws.amazon.com/blogs/compute/introducing-mutual-tls-authentication-for-amazon-msk-as-an-event-source). + * If you use authentication, for **Authentication method** , choose the authentication method to use. @@ -98 +82 @@ For instructions on how to generate a client certificate, see [ Introducing mutu -Amazon MSK doesn't support self-signed server certificates, because all brokers in Amazon MSK use [public certificates](https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.html) signed by [Amazon Trust Services CAs](https://www.amazontrust.com/repository/), which Lambda trusts by default. + * If you use authentication, for **Secrets Manager key** , choose the Secrets Manager key that contains the authentication credentials needed to access your cluster. @@ -100 +84 @@ Amazon MSK doesn't support self-signed server certificates, because all brokers -For more information about mTLS for Amazon MSK, see [Mutual TLS Authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html) in the _Amazon Managed Streaming for Apache Kafka Developer Guide_. + 7. Under **Event poller configuration** , make the necessary configurations. @@ -102 +86 @@ For more information about mTLS for Amazon MSK, see [Mutual TLS Authentication]( -### Configuring the mTLS secret + * Choose **Activate trigger** to enable the trigger immediately after creation. @@ -104 +88 @@ For more information about mTLS for Amazon MSK, see [Mutual TLS Authentication]( -The CLIENT_CERTIFICATE_TLS_AUTH secret requires a certificate field and a private key field. For an encrypted private key, the secret requires a private key password. Both the certificate and private key must be in PEM format. + * Choose whether you want to **Configure provisioned mode** for your event source mapping. For more information, see Event poller scaling modes. @@ -106 +90 @@ The CLIENT_CERTIFICATE_TLS_AUTH secret requires a certificate field and a privat -###### Note + * If you configure provisioned mode, enter a value for **Minimum event pollers** , a value for **Maximum event pollers** , or both values. @@ -108 +92 @@ The CLIENT_CERTIFICATE_TLS_AUTH secret requires a certificate field and a privat -Lambda supports the [PBES1](https://datatracker.ietf.org/doc/html/rfc2898/#section-6.1) (but not PBES2) private key encryption algorithms. + * For **Starting position** , choose how you want Lambda to start reading from your stream. For more information, see Polling and stream starting positions. @@ -110 +94 @@ Lambda supports the [PBES1](https://datatracker.ietf.org/doc/html/rfc2898/#secti -The certificate field must contain a list of certificates, beginning with the client certificate, followed by any intermediate certificates, and ending with the root certificate. Each certificate must start on a new line with the following structure: + 8. Under **Batching** , make the necessary configurations. For more information about batching, see [Batching behavior](./invocation-eventsourcemapping.html#invocation-eventsourcemapping-batching). @@ -111,0 +96 @@ The certificate field must contain a list of certificates, beginning with the cl + 1. For **Batch size** , enter the maximum number of messages to receive in a single batch. @@ -113,3 +98 @@ The certificate field must contain a list of certificates, beginning with the cl - -----BEGIN CERTIFICATE----- - <certificate contents> - -----END CERTIFICATE----- + 2. For **Batch window** , enter the maximum number of seconds that Lambda spends gathering records before invoking the function. @@ -117 +100 @@ The certificate field must contain a list of certificates, beginning with the cl -Secrets Manager supports secrets up to 65,536 bytes, which is enough space for long certificate chains. + 9. Under **Filtering** , make the necessary configurations. For more information about filtering, see [Using event filtering with an Amazon MSK event source](./with-msk-filtering.html). @@ -119 +102 @@ Secrets Manager supports secrets up to 65,536 bytes, which is enough space for l -The private key must be in [PKCS #8](https://datatracker.ietf.org/doc/html/rfc5208) format, with the following structure: + * For **Filter criteria** , add filter criteria definitions to determine whether or not to process an event. @@ -120,0 +104 @@ The private key must be in [PKCS #8](https://datatracker.ietf.org/doc/html/rfc52 + 10. Under **Failure handling** , make the necessary configurations. For more information about failure handling, see [Capturing discarded batches for an Amazon MSK event source](./with-msk-on-failure.html). @@ -122,3 +106 @@ The private key must be in [PKCS #8](https://datatracker.ietf.org/doc/html/rfc52 - -----BEGIN PRIVATE KEY----- - <private key contents> - -----END PRIVATE KEY----- + * For **On-failure destination** , specify the ARN of your on-failure destination. @@ -126 +108 @@ The private key must be in [PKCS #8](https://datatracker.ietf.org/doc/html/rfc52 -For an encrypted private key, use the following structure: + 11. For **Tags** , enter the tags to associate with this event source mapping. @@ -127,0 +110 @@ For an encrypted private key, use the following structure: + 12. To create the trigger, choose **Add**. @@ -129,3 +111,0 @@ For an encrypted private key, use the following structure: - -----BEGIN ENCRYPTED PRIVATE KEY----- - <private key contents> - -----END ENCRYPTED PRIVATE KEY----- @@ -133 +112,0 @@ For an encrypted private key, use the following structure: -The following example shows the contents of a secret for mTLS authentication using an encrypted private key. For an encrypted private key, you include the private key password in the secret. @@ -136,21 +115 @@ The following example shows the contents of a secret for mTLS authentication usi - { - "privateKeyPassword": "testpassword", - "certificate": "-----BEGIN CERTIFICATE----- - MIIE5DCCAsygAwIBAgIRAPJdwaFaNRrytHBto0j5BA0wDQYJKoZIhvcNAQELBQAw - ... - j0Lh4/+1HfgyE2KlmII36dg4IMzNjAFEBZiCRoPimO40s1cRqtFHXoal0QQbIlxk - cmUuiAii9R0= - -----END CERTIFICATE----- - -----BEGIN CERTIFICATE----- - MIIFgjCCA2qgAwIBAgIQdjNZd6uFf9hbNC5RdfmHrzANBgkqhkiG9w0BAQsFADBb - ... - rQoiowbbk5wXCheYSANQIfTZ6weQTgiCHCCbuuMKNVS95FkXm0vqVD/YpXKwA/no - c8PH3PSoAaRwMMgOSA2ALJvbRz8mpg== - -----END CERTIFICATE-----", - "privateKey": "-----BEGIN ENCRYPTED PRIVATE KEY----- - MIIFKzBVBgkqhkiG9w0BBQ0wSDAnBgkqhkiG9w0BBQwwGgQUiAFcK5hT/X7Kjmgp - ... - QrSekqF+kWzmB6nAfSzgO9IaoAaytLvNgGTckWeUkWn/V0Ck+LdGUXzAC4RxZnoQ - zp2mwJn2NYB7AZ7+imp0azDZb+8YG2aUCiyqb6PnnA==