AWS powertools documentation change
Summary
Added comprehensive documentation for processing messages from Amazon MSK/self-managed Apache Kafka with AWS Lambda Powertools batch processing utility, including CloudFormation template, IAM permissions, code examples, and event structure details
Security assessment
The change adds documentation for Kafka/MSK integration with Lambda batch processing, including IAM permissions configuration for Kafka cluster access and SQS dead-letter queue permissions. While this includes security-related IAM policies (kafka:DescribeCluster, kafka-cluster:ReadData, etc.), there is no evidence this addresses a specific security vulnerability or incident. The change primarily documents a new feature integration.
Diff
diff --git a/powertools/python/latest/utilities/batch.md b/powertools/python/latest/utilities/batch.md index 2c3a6f778..31afa6e8d 100644 --- a//powertools/python/latest/utilities/batch.md +++ b//powertools/python/latest/utilities/batch.md @@ -57,0 +58 @@ Metrics + * Processing messages from Kafka @@ -84,0 +86 @@ Metrics + * [ Metadata ](../metadata/) @@ -181,0 +184 @@ Table of contents + * Processing messages from Kafka @@ -206 +209 @@ Table of contents -The batch processing utility handles partial failures when processing batches from Amazon SQS, Amazon Kinesis Data Streams, and Amazon DynamoDB Streams. +The batch processing utility handles partial failures when processing batches from Amazon SQS, Amazon Kinesis Data Streams, Amazon DynamoDB Streams, and Amazon MSK/self-managed Apache Kafka. @@ -211 +214 @@ The batch processing utility handles partial failures when processing batches fr - BatchSource: Amazon SQS <br/><br/> Amazon Kinesis Data Streams <br/><br/> Amazon DynamoDB Streams <br/><br/> + BatchSource: Amazon SQS <br/><br/> Amazon Kinesis Data Streams <br/><br/> Amazon DynamoDB Streams <br/><br/> Amazon MSK / Apache Kafka <br/><br/> @@ -242 +245 @@ The batch processing utility handles partial failures when processing batches fr -When using SQS, Kinesis Data Streams, or DynamoDB Streams as a Lambda event source, your Lambda functions are triggered with a batch of messages. +When using SQS, Kinesis Data Streams, DynamoDB Streams, or Amazon MSK/Apache Kafka as a Lambda event source, your Lambda functions are triggered with a batch of messages. @@ -256,0 +260 @@ This behavior changes when you enable Report Batch Item Failures feature in your + * **Kafka (MSK and self-managed)**. Failed records are identified by topic-partition and offset. Only failed records will be retried. @@ -262 +266 @@ We recommend implementing processing logic in an [idempotent manner](../idempote -You can find more details on how Lambda works with either [SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html), [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html), or [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html) in the AWS Documentation. +You can find more details on how Lambda works with either [SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html), [Kinesis](https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html), [DynamoDB](https://docs.aws.amazon.com/lambda/latest/dg/with-ddb.html), or [MSK/Kafka](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html) in the AWS Documentation. @@ -276 +280 @@ You do not need any additional IAM permissions to use this utility, except for w -SQSKinesis Data StreamsDynamoDB Streams +SQSKinesis Data StreamsDynamoDB StreamsKafka (MSK) @@ -637,0 +642,184 @@ template.yaml +template.yaml +--- + + + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + 11 + 12 + 13 + 14 + 15 + 16 + 17 + 18 + 19 + 20 + 21 + 22 + 23 + 24 + 25 + 26 + 27 + 28 + 29 + 30 + 31 + 32 + 33 + 34 + 35 + 36 + 37 + 38 + 39 + 40 + 41 + 42 + 43 + 44 + 45 + 46 + 47 + 48 + 49 + 50 + 51 + 52 + 53 + 54 + 55 + 56 + 57 + 58 + 59 + 60 + 61 + 62 + 63 + 64 + 65 + 66 + 67 + 68 + 69 + 70 + 71 + 72 + 73 + 74 + 75 + 76 + 77 + 78 + 79 + 80 + 81 + 82 + 83 + 84 + 85 + 86 + 87 + +| + + + AWSTemplateFormatVersion: "2010-09-09" + Transform: AWS::Serverless-2016-10-31 + Description: Kafka/MSK partial batch response sample + + Globals: + Function: + Timeout: 30 + MemorySize: 256 + Runtime: python3.12 + Tracing: Active + Environment: + Variables: + POWERTOOLS_LOG_LEVEL: INFO + POWERTOOLS_SERVICE_NAME: kafka-processor + + Parameters: + MSKClusterArn: + Type: String + Description: ARN of the MSK cluster + KafkaTopic: + Type: String + Description: Name of the Kafka topic to consume from + Default: mytopic + + Resources: + KafkaConsumerFunction: + Type: AWS::Serverless::Function + Properties: + Handler: app.lambda_handler + CodeUri: hello_world + Policies: + # Permissions for MSK + - Version: "2012-10-17" + Statement: + - Effect: "Allow" + Action: + - kafka:DescribeCluster + - kafka:DescribeClusterV2 + - kafka:GetBootstrapBrokers + Resource: !Ref MSKClusterArn + - Effect: "Allow" + Action: + - kafka-cluster:Connect + - kafka-cluster:DescribeGroup + - kafka-cluster:AlterGroup + - kafka-cluster:DescribeTopic + - kafka-cluster:ReadData + - kafka-cluster:DescribeClusterDynamicConfiguration + Resource: + - !Ref MSKClusterArn + - !Sub "${MSKClusterArn}/*" + # Lambda Destinations require additional permissions + # to send failure records to DLQ + - Version: "2012-10-17" + Statement: + Effect: "Allow" + Action: + - sqs:GetQueueAttributes + - sqs:GetQueueUrl + - sqs:SendMessage + Resource: !GetAtt SampleDLQ.Arn + Events: + MSKEvent: + Type: MSK + Properties: + Stream: !Ref MSKClusterArn + Topics: + - !Ref KafkaTopic + StartingPosition: LATEST + BatchSize: 100 + MaximumBatchingWindowInSeconds: 5 + DestinationConfig: + OnFailure: + Destination: !GetAtt SampleDLQ.Arn + FunctionResponseTypes: + - ReportBatchItemFailures +