AWS lambda documentation change
Summary
Added sections for advanced error handling features and cost optimization strategies in Kafka provisioned mode, including retry configurations, failure destinations, EPU utilization, and ESM grouping capabilities
Security assessment
The changes focus on operational improvements (error handling, performance tuning, cost optimization) rather than addressing security vulnerabilities or introducing security controls. While failure destinations and retry configurations improve reliability, there is no mention of authentication, encryption, access control, or other security mechanisms. The network settings note about AWS PrivateLink removal appears in existing content rather than the new changes.
Diff
diff --git a/lambda/latest/dg/kafka-scaling-modes.md b/lambda/latest/dg/kafka-scaling-modes.md index 484602d62..4f901f7be 100644 --- a//lambda/latest/dg/kafka-scaling-modes.md +++ b//lambda/latest/dg/kafka-scaling-modes.md @@ -5 +5 @@ -On-demand mode (default)Provisioned modeBest practices +On-demand mode (default)Provisioned modeAdvanced error handling and performance featuresBest practicesCost Optimization for Provisioned mode @@ -80,0 +81,11 @@ To disable provisioned mode and return to default (on-demand) mode, you can use +## Advanced error handling and performance features + +For Kafka event source mappings with provisioned mode enabled, you can configure additional features to improve error handling and performance: + + * [Retry configurations](./kafka-retry-configurations.html) – Control how Lambda handles failed records with maximum retry attempts, record age limits, batch splitting, and partial batch responses. + + * [Kafka on-failure destinations](./kafka-on-failure-destination.html) – Send failed records to a Kafka topic for later processing or analysis. + + + + @@ -90,0 +102,133 @@ When activating provisioned mode, update your network settings to remove AWS Pri +## Cost Optimization for Provisioned mode + +### Provisioned mode pricing + +Provisioned mode is charged based on the provisioned minimum event pollers, and the event pollers consumed during autoscaling. Charges are calculated using a billing unit called Event Poller Unit (EPU). You pay for the number and duration of EPUs used, measured in Event-Poller-Unit-hours. You can use Provisioned mode with either a single ESM for performance-sensitive applications or you can group multiple ESMs within the same VPC to share EPU capacity and costs. We will deep dive on two capabilities that help you optimize your Provisioned mode costs. For pricing details, see [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/). + +### Enhanced EPU Utilization + +Each EPU supports up to 20 MB/s throughput capacity for event polling and supports a default of 10 event pollers. When you create a Provisioned mode for Kafka ESM by setting minimum and maximum pollers, it uses minimum poller number to provision EPUs based on default of 10 event pollers per EPU. However, each event poller can independently scale to support up to 5 MB/s of throughput capacity, which may require lower density of event pollers on a specific EPU and can trigger scaling of EPUs. The number of event pollers allocated on an EPU depends on the compute capacity consumed by each event poller. This approach of enhanced EPU utilization allows event pollers with varying throughput requirements to utilize EPU capacity effectively, reducing costs for all ESMs. + +### ESM grouping + +To optimize your Provisioned mode costs further, you can group multiple Kafka ESMs to share EPU capacity. With ESM grouping and enhanced EPU Utilization, you can reduce your Provisioned mode costs up to 90% for low-throughput workloads compared to running in single ESM mode. All ESMs that require less than 1 EPU capacity will benefit from ESM grouping. Such ESMs typically require few minimum event pollers to support their throughput needs. This capability will allow you to adopt Provisioned mode for all your Kafka workloads, and benefit from features like schema validation, filtering of Avro/Protobuf events, low-latency invocations, and enhanced error handling that only available in Provisioned mode. + +When you configure the `PollerGroupName` parameter with the same value for multiple ESMs within the same Amazon VPC, those ESMs share EPU resources instead of each requiring dedicated EPU capacity. You can group up to 100 ESMs per poller group and aggregate maximum pollers across all ESMs in a group cannot exceed 2000. + +#### To configure ESM grouping (console) + + 1. Open the [Functions page](https://console.aws.amazon.com/lambda/home#/functions) of the Lambda console. + + 2. Choose your function. + + 3. Choose **Configuration** , and then choose **Triggers**. + + 4. When creating a new Kafka event source mapping, or editing an existing one, select **Configure** under **Provisioned mode**. + + 5. For **Minimum event pollers** , enter a value between 1 and 200. + + 6. For **Maximum event pollers** , enter a value between 1 and 2,000. + + 7. For **Poller group name** , enter an identifier for the group. Use the same name for other ESMs you want to group together. + + 8. Choose **Save**. + + + + +#### To configure ESM grouping (AWS CLI) + +The following example creates an ESM with a poller group named `production-app-group`: + + + aws lambda create-event-source-mapping \ + --function-name myFunction1 \ + --event-source-arn arn:aws:kafka:us-east-1:123456789012:cluster/MyCluster/abcd1234 \ + --topics topic1 \ + --starting-position LATEST \ + --provisioned-poller-config '{ + "MinimumPollers": 1, + "MaximumPollers": 10, + "PollerGroupName": "production-app-group" + }' + +To add another ESM to the same group (sharing EPU capacity), use the same PollerGroupName: + + + aws lambda create-event-source-mapping \ + --function-name myFunction2 \ + --event-source-arn arn:aws:kafka:us-east-1:123456789012:cluster/MyCluster/abcd1234 \ + --topics topic2 \ + --starting-position LATEST \ + --provisioned-poller-config '{ + "MinimumPollers": 1, + "MaximumPollers": 10, + "PollerGroupName": "production-app-group" + }' + +###### Note + +You can update the `PollerGroupName` to move an ESM to a different group, or remove an ESM from a group by passing an empty string ("") for `PollerGroupName`: + + + # Move ESM to a different group + aws lambda update-event-source-mapping \ + --uuid a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \ + --provisioned-poller-config '{ + "MinimumPollers": 1, + "MaximumPollers": 10, + "PollerGroupName": "new-group-name" + }' + + # Remove ESM from group (use dedicated resources) + aws lambda update-event-source-mapping \ + --uuid a1b2c3d4-5678-90ab-cdef-EXAMPLE11111 \ + --provisioned-poller-config '{ + "MinimumPollers": 1, + "MaximumPollers": 10, + "PollerGroupName": "" + }' + +#### Grouping strategy considerations + + * **Application boundary** – Group ESMs that belong to the same applications or services for better cost allocation and management. Consider using naming conventions like `app-name-environment` (e.g., `order-processor-prod`). + + * **Traffic pattern** – Avoid grouping ESMs with high throughput and spiky traffic pattern, as this may lead to resource contention. + + * **Blast radius** – Consider the impact if the shared infrastructure experiences issues. All ESMs in the same group are affected by shared resource limitations. For mission-critical workloads, you may want to use separate groups or dedicated ESMs. + + + + +#### Cost optimization example + +Consider a scenario where you have 10 ESMs, each configured with 1 event poller and throughput under 2 MB/s: + +**Without grouping:** + + * Each ESM requires its own EPU + + * Total EPUs needed: 10 + + * Cost per EPU: $0.185/hour in US East (N. Virginia) + + * Monthly EPU cost (720 hours): 10 × 720 × $0.185 = $1,332 + + + + +**With grouping:** + + * All 10 ESMs share EPU capacity + + * 10 event pollers fit in 1 EPU (with new 10 poller per EPU support) + + * Total EPUs needed: 1 + + * Monthly EPU cost (720 hours): 1 × 720 × $0.185 = $133.20 + + * **Cost savings: 90%** ($1,198.80 savings per month) + + + + @@ -99 +243 @@ Configuration parameters -Consumer group ID +Polling and stream positions