AWS secretsmanager documentation change
Summary
Restructured troubleshooting guide with new sections about concurrency issues, added detailed steps for Lambda rotation failures, and expanded error resolution guidance
Security assessment
The changes improve documentation around secret rotation reliability but don't address specific vulnerabilities. However, proper rotation implementation is a security best practice. Added warnings about concurrency settings help prevent rotation failures that could leave secrets unchanged longer than intended.
Diff
diff --git a/secretsmanager/latest/userguide/troubleshoot_rotation.md b/secretsmanager/latest/userguide/troubleshoot_rotation.md index 578e09b0c..a4fd7d04b 100644 --- a//secretsmanager/latest/userguide/troubleshoot_rotation.md +++ b//secretsmanager/latest/userguide/troubleshoot_rotation.md @@ -5 +5 @@ -No activity after "Found credentials in environment variables"No activity after "createSecret"Error: "Access to KMS is not allowed"Error: "Key is missing from secret JSON" Error: "setSecret: Unable to log into database"Error: "Unable to import module 'lambda_function'" Upgrade an existing rotation function from Python 3.7 to 3.9AWS Lambda secret rotation with PutSecretValue failed +Troubleshooting stepsNo activity after "Found credentials in environment variables"No activity after "createSecret"Error: "Access to KMS is not allowed"Error: "Key is missing from secret JSON" Error: "setSecret: Unable to log into database"Error: "Unable to import module 'lambda_function'" Upgrade an existing rotation function from Python 3.7 to 3.9AWS Lambda secret rotation with PutSecretValue failedIntermittent rotation failures @@ -15 +15 @@ Some services can manage secrets for you, including managing automatic rotation. -###### To view the CloudWatch logs for your Lambda function +###### Topics @@ -17,12 +17 @@ Some services can manage secrets for you, including managing automatic rotation. - 1. Open the Secrets Manager console at [https://console.aws.amazon.com/secretsmanager/](https://console.aws.amazon.com/secretsmanager/). - - 2. Choose your secret, and then on the details page, under **Rotation configuration** , choose the Lambda rotation function. The Lambda console opens. - - 3. On the **Monitor** tab, choose **Logs** , and then choose **View logs in CloudWatch**. - -The CloudWatch console opens and displays the logs for your function. - - - - -###### To interpret the logs + * How to troubleshoot secret rotation failures in AWS Lambda functions @@ -45,0 +35,64 @@ The CloudWatch console opens and displays the logs for your function. + * Error: "Error when executing lambda <arn> during <a rotation> step" + + + + +## How to troubleshoot secret rotation failures in AWS Lambda functions + +If you're experiencing secret rotation failures with your Lambda functions, use the following steps to troubleshoot and resolve the issue. + +### Possible causes + + * Insufficient concurrent executions for the Lambda function + + * Race conditions due to multiple API calls during rotation + + * Incorrect Lambda function logic + + * Networking issues between the Lambda function and the database + + + + +### General troubleshooting steps + + 1. Analyze CloudWatch logs: + + * Look for specific error messages or unexpected behavior in the Lambda function logs + + * Verify that all rotation steps (**CreateSecret** , **SetSecret** , **TestSecret** , **FinishSecret**) are being attempted + + 2. Review API calls during rotation: + + * Avoid making mutating API calls on the secret during Lambda rotation + + * Ensure there's no race condition between **RotateSecret** and **PutSecretValue** calls + + 3. Verify Lambda function logic: + + * Confirm you're using the latest AWS sample code for secret rotation + + * If using custom code, review it for proper handling of all rotation steps + + 4. Check network configuration: + + * Verify security group rules allow the Lambda function to access the database + + * Ensure proper VPC endpoint or public endpoint access for Secrets Manager + + 5. Test secret versions: + + * Verify that the AWSCURRENT version of the secret allows database access + + * Check if AWSPREVIOUS or AWSPENDING versions are valid + + 6. Clear pending rotations: + + * If rotation consistently fails, clear the AWSPENDING staging label and retry rotation + + 7. Check Lambda concurrency settings: + + * Verify that concurrency settings are appropriate for your workload + + * If you suspect concurrency issues, see the "Troubleshooting concurrency-related rotation failures" section + @@ -294 +347 @@ If you used the AWS CDK prior to version v2.94.0 to set up rotation for your sec -If you use an assumed role or a cross-account rotation with Secrets Manager and you find a RotationFailed event in AWS CloudTrail with the message: **`Pending secret version`VERSION_ID `for Secret `SECRET_ARN ` was not created by Lambda `LAMBDA_ARN.` Remove the `AWSPENDING` staging label and restart rotation`**, then you need to update your Lambda function to use the `RotationToken` parameter. +If you use an assumed role or a cross-account rotation with Secrets Manager and you find a **RotationFailed** event in CloudTrail with the message: **`Pending secret version`VERSION_ID `for Secret `SECRET_ARN ` was not created by Lambda `LAMBDA_ARN.` Remove the `AWSPENDING` staging label and restart rotation`**, then you need to update your Lambda function to use the `RotationToken` parameter. @@ -425,0 +479,39 @@ After updating your Lambda function code, [upload it to rotate your secret](http +## Error: "Error when executing lambda `<arn>` during `<a rotation>` step" + +If you're experiencing intermittent secret rotation failures with your Lambda function getting stuck in a loop of sets, for example between **CreateSecret** and **SetSecret** , the issue may be related to concurrency settings. + +### Concurrency troubleshooting steps + +###### Warning + +Setting the provisioned concurrency parameter to a value lower than 10 can cause throttling due to insufficient execution threads for the Lambda function. For more information, see [ Understanding reserved concurrency and provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html#reserved-and-provisioned) in the AWS Lambda AWS Lambda Developer Guide. + + 1. Check and adjust Lambda concurrency settings: + + * Verify that `reserved_concurrent_executions` is not set too low (for example, 1) + + * If using reserved concurrency, set it to at least 10 + + * Consider using unreserved concurrency for more flexibility + + 2. For provisioned concurrency: + + * Don't set the provisioned concurrency parameter explicitly (for example, in Terraform). + + * If you must set it, use a value of at least 10. + + * Test thoroughly to make sure the chosen value works for your use case. + + 3. Monitor and adjust concurrency: + + * Calculate concurrency using this formula: Concurrency = (average requests per second) * (average request duration in seconds). For more information, see [Estimating reserved concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html#estimating-reserved-concurrency). + + * Observe and record values during rotations to determine the appropriate concurrency settings. + + * Be careful when setting low concurrency values. They can cause throttling if there aren't enough available execution threads. + + + + +For more information on configuring Lambda concurrency, see [Configuring reserved concurrency](https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html) and [Configuring provisioned concurrency](https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html) in the AWS Lambda Developer Guide. +