AWS bedrock documentation change
Summary
Updated reward function documentation with expanded Lambda implementation details, added input/output structures, design guidelines, and clarified security requirements for custom Lambda functions.
Security assessment
The changes add security documentation by specifying required Lambda execution role permissions and timeout configurations. The new 'Design guidelines' section includes security best practices like evaluating safety and maintaining stable scoring. However, there's no evidence of addressing a specific security vulnerability.
Diff
diff --git a/bedrock/latest/userguide/reward-functions.md b/bedrock/latest/userguide/reward-functions.md index 280b47994..84dda3599 100644 --- a//bedrock/latest/userguide/reward-functions.md +++ b//bedrock/latest/userguide/reward-functions.md @@ -5 +5 @@ -Reinforcement Learning via Verifiable Rewards (RLVR)Reinforcement Learning via AI Feedback (RLAIF) +Reinforcement Learning via Verifiable Rewards (RLVR)Reinforcement Learning via AI Feedback (RLAIF)Lambda function implementation details @@ -7 +7 @@ Reinforcement Learning via Verifiable Rewards (RLVR)Reinforcement Learning via A -# Setting up reward functions +# Setting up reward functions for Amazon Nova models @@ -9 +9 @@ Reinforcement Learning via Verifiable Rewards (RLVR)Reinforcement Learning via A -Reward functions evaluate response quality and provide feedback signals for model training. Choose the approach that matches your task requirements. +Reward functions evaluate response quality and provide feedback signals for model training. You can set up reward functions using custom Lambda functions or Amazon Bedrock-hosted foundation models as judges. Guided templates are available to simplify reward function creation for common tasks like instruction following and format validation. Choose the approach that matches your task requirements. @@ -13 +13 @@ Reward functions evaluate response quality and provide feedback signals for mode -RLVR enables you to optimize models for objective tasks such as code generation or math reasoning. You can define reward functions using verifiable rule-based graders or use ready-to-use templates for common use cases such as format checks, summarization, and text similarity. +RLVR optimizes models for objective tasks such as code generation or math reasoning using verifiable rule-based graders or ready-to-use templates. @@ -17 +17 @@ You have two options for RLVR (Custom Code): - * Use console-provided templates - The Amazon Bedrock console provides sample templates for grader Lambda functions: +Amazon Bedrock console provides sample templates for grader Lambda functions: @@ -23 +23 @@ You have two options for RLVR (Custom Code): - * Generic grader Lambda template with boilerplate code for your grader Lambda function + * Generic grader Lambda template with boilerplate code @@ -25 +24,0 @@ You have two options for RLVR (Custom Code): -Before setting up your Lambda function, follow the instructions in the provided template on the **Create RFT job** page in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock). @@ -27 +26,13 @@ Before setting up your Lambda function, follow the instructions in the provided - * Bring your own Lambda function — Create custom reward functions executed through Lambda functions using your own Lambda ARN. You can combine multiple graders to produce a single score. + + +Follow the instructions in the provided template on the **Create RFT job** page in the [Amazon Bedrock console](https://console.aws.amazon.com/bedrock). + +Create custom reward functions using your own Lambda ARN for complex logic, external APIs, multi-step calculations, or combining multiple evaluation criteria. + +###### Note + +If you bring your own Lambda function, keep the following in mind: + + * Increase the Lambda timeout from default 3 seconds to maximum 15 minutes for complex evaluations. + + * The Lambda execution role needs permissions to invoke models as described in [Access and security for Amazon Nova models](./rft-access-security.html). @@ -34 +45 @@ Before setting up your Lambda function, follow the instructions in the provided -RLAIF enables optimization for subjective tasks such as instruction following or chatbot interactions. You can use AI-based judges with ready-to-use templates for common use cases to evaluate response quality based on criteria you define. +RLAIF optimizes models for subjective tasks such as instruction following or chatbot interactions using AI-based judges with ready-to-use templates. @@ -47 +58 @@ RLAIF enables optimization for subjective tasks such as instruction following or -You can use the LLM-as-Judge prompt templates provided in the Amazon Bedrock console: +Available LLM-as-Judge prompt templates in the Amazon Bedrock console: @@ -62 +73,57 @@ You can use the LLM-as-Judge prompt templates provided in the Amazon Bedrock con - * When you use the console's **Model as Judge** option, Amazon Bedrock automatically converts your configuration into a Lambda function that executes during training. +The console's **Model as Judge** option automatically converts your configuration into a Lambda function during training. + +## Lambda function implementation details + +When implementing custom Lambda reward functions, your function must accept and return data in the following format. + +Input structure + + + + [{ + "id": "123", + "messages": [ + { + "role": "user", + "content": "Do you have a dedicated security team?" + }, + { + "role": "assistant", + "content": "As an AI developed by Amazon, I don not have a dedicated security team..." + } + ], + "metadata": { + "reference_answer": { + "compliant": "No", + "explanation": "As an AI developed by Company, I do not have a traditional security team..." + }, + "my_key": "sample-001" + } + }] + +Output structure + + + + [{ + "id": "123", + "aggregate_reward_score": 0.85, + "metrics_list": [ + { + "name": "accuracy", + "value": 0.9, + "type": "Reward" + }, + { + "name": "policy_compliance", + "value": 0.8, + "type": "Metric" + } + ] + }] + +**Design guidelines** + + * **Rank responses** – Give the best answer a clearly higher score + + * **Use consistent checks** – Evaluate task completion, format adherence, safety, and reasonable length @@ -64 +131 @@ You can use the LLM-as-Judge prompt templates provided in the Amazon Bedrock con - * If you bring your own Lambda function, the Lambda execution role needs necessary permissions to invoke models with model ID or inference profile as described in [Grader Lambda function permissions for RLAIF](./rft-access-security.html#rft-bedrock-permissions). + * **Maintain stable scaling** – Keep scores normalized and non-exploitable @@ -75 +142 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Requirements for training data sources +Prepare data @@ -77 +144 @@ Requirements for training data sources -Create a reinforcement fine-tuning job +Create fine-tuning jobs