AWS bedrock documentation change
Summary
Restructured and expanded reinforcement fine-tuning documentation with new sections on applications, benefits, supported models, best practices, and workflow automation. Added specific use cases, performance metrics, and additional supported models.
Security assessment
The changes are organizational and content enhancements without addressing specific vulnerabilities. The removed 'security and compliance' bullet was re-added verbatim in the benefits section, indicating no security issue was fixed. New content focuses on workflow optimization and use cases rather than security vulnerabilities or weaknesses.
Diff
diff --git a/bedrock/latest/userguide/reinforcement-fine-tuning.md b/bedrock/latest/userguide/reinforcement-fine-tuning.md index 63edcff28..832a07056 100644 --- a//bedrock/latest/userguide/reinforcement-fine-tuning.md +++ b//bedrock/latest/userguide/reinforcement-fine-tuning.md @@ -5 +5 @@ -Supported models for reinforcement fine-tuningHow reinforcement fine-tuning works +Reinforcement fine-tuning applications and scenariosBenefits of reinforcement fine-tuningSupported models for reinforcement fine-tuningHow reinforcement fine-tuning worksReinforcement fine-tuning best practices @@ -9 +9 @@ Supported models for reinforcement fine-tuningHow reinforcement fine-tuning work -Reinforcement fine-tuning is a model customization technique in Amazon Bedrock. It improves foundation model performance by teaching models what constitutes a "good" response through feedback signals called rewards. While traditional fine-tuning methods depend on labeled datasets, reinforcement fine-tuning uses a feedback-driven approach. This allows models to improve iteratively based on reward signals. Instead of learning from fixed examples, it uses reward functions to evaluate and judge which responses are considered good for particular business use cases. +Reinforcement fine-tuning is a model customization technique in Amazon Bedrock that improves foundation model performance by teaching models what constitutes a "good" response through feedback signals called rewards. Unlike traditional fine-tuning methods that depend on labeled datasets, reinforcement fine-tuning uses a feedback-driven approach that iteratively optimizes the model to maximize these rewards. @@ -11 +11 @@ Reinforcement fine-tuning is a model customization technique in Amazon Bedrock. -Reinforcement fine-tuning teaches models to understand what makes a quality response. You don't need massive amounts of pre-labeled training data. This makes advanced model customization in Amazon Bedrock more accessible and cost-effective. +## Reinforcement fine-tuning applications and scenarios @@ -13 +13 @@ Reinforcement fine-tuning teaches models to understand what makes a quality resp -The capability supports two approaches to provide flexibility for optimizing models: +Use reinforcement fine-tuning when you can define clear, measurable success criteria for evaluating response quality. Reinforcement fine-tuning excels in domains where output quality can be objectively measured, especially when multiple valid responses exist or when optimal responses are difficult to define upfront. It's ideal for: @@ -15 +15 @@ The capability supports two approaches to provide flexibility for optimizing mod - * **Reinforcement Learning with Verifiable Rewards (RLVR)** \- Uses rule-based graders for objective tasks like code generation or math reasoning + * Mathematical problem-solving and code generation (using rule-based graders for objective evaluation) @@ -17 +17 @@ The capability supports two approaches to provide flexibility for optimizing mod - * **Reinforcement Learning from AI Feedback (RLAIF)** \- Uses AI-based judges for subjective tasks like instruction following or content moderation + * Scientific reasoning and structured data analysis @@ -18,0 +19 @@ The capability supports two approaches to provide flexibility for optimizing mod + * Subjective tasks like instruction following, content moderation, and creative writing (using AI-based judges) @@ -19,0 +21 @@ The capability supports two approaches to provide flexibility for optimizing mod + * Tasks requiring step-by-step reasoning or multi-turn problem solving @@ -20,0 +23 @@ The capability supports two approaches to provide flexibility for optimizing mod + * Scenarios with multiple valid solutions where some are clearly better than others @@ -22 +25 @@ The capability supports two approaches to provide flexibility for optimizing mod -For more information, see [Setting up reward functions](./reward-functions.html). + * Applications balancing multiple objectives (accuracy, efficiency, style) @@ -24 +27 @@ For more information, see [Setting up reward functions](./reward-functions.html) -Reinforcement fine-tuning can provide the following benefits: + * Applications requiring iterative improvement, personalization, or adherence to complex business rules @@ -26 +29 @@ Reinforcement fine-tuning can provide the following benefits: - * **Improved model performance** \- Reinforcement fine-tuning improves model accuracy compared to base models. This enables optimization for price and performance by training smaller, faster, and more efficient model variants. + * Scenarios where success can be verified programmatically through execution results or performance metrics @@ -28 +31 @@ Reinforcement fine-tuning can provide the following benefits: - * **Flexible training data** \- Amazon Bedrock automates much of the complexity. This makes reinforcement fine-tuning accessible to developers building AI applications. You can easily train models using existing Amazon Bedrock model invocation logs as training data or upload your datasets. + * Cases where collecting high-quality labeled examples is expensive or impractical @@ -30 +32,0 @@ Reinforcement fine-tuning can provide the following benefits: - * **Security and compliance** \- Your proprietary data never leaves AWS's secure, governed environment during the customization process. @@ -33,0 +36 @@ Reinforcement fine-tuning can provide the following benefits: +## Benefits of reinforcement fine-tuning @@ -35,5 +38 @@ Reinforcement fine-tuning can provide the following benefits: -###### Topics - - * Supported models for reinforcement fine-tuning - - * How reinforcement fine-tuning works + * **Improved model performance** – Reinforcement fine-tuning improves model accuracy by up to 66% on average compared to base models. This enables optimization for price and performance by fine-tuning smaller, faster, and more efficient model variants. @@ -41 +40 @@ Reinforcement fine-tuning can provide the following benefits: - * [Reinforcement fine-tuning access and security](./rft-access-security.html) + * **Ease of use** – Amazon Bedrock automates the complexity of reinforcement fine-tuning, making it accessible to developers building AI applications. You can fine-tune models using your uploaded datasets or existing API invocation logs. You can define reward functions that grade model outputs with custom code using Lambda or model-as-a-judge grader, with built-in templates that help with quick setup. @@ -43,3 +42 @@ Reinforcement fine-tuning can provide the following benefits: - * [Prepare your training data and reward functions for reinforcement fine-tuning](./rft-prepare-data.html) - - * [Create a reinforcement fine-tuning job](./rft-submit-job.html) + * **Security and compliance** – Your proprietary data never leaves AWS's secure, governed environment during the customization process. @@ -54,3 +51,5 @@ The following table shows the foundation models that you can customize with rein -Supported models for reinforcement fine-tuning Provider | Model | Model ID | Single-region model support ----|---|---|--- -Amazon | Nova 2 Lite | amazon.nova-2-lite-v1:0:256k | us-east-1 +Supported models for reinforcement fine-tuning Provider | Model | Model ID | Region name | Region +---|---|---|---|--- +Amazon | Nova 2 Lite | amazon.nova-2-lite-v1:0:256k | US East (N. Virginia) | us-east-1 +OpenAI | gpt-oss-20B | openai.gpt-oss-20b | US West (Oregon) | us-west-2 +Qwen | Qwen3 32B | qwen.qwen3-32b | US West (Oregon) | us-west-2 @@ -60 +59,3 @@ Amazon | Nova 2 Lite | amazon.nova-2-lite-v1:0:256k | us-east-1 -Amazon Bedrock fully automates the RFT workflow through a three-stage process: +Amazon Bedrock fully automates the reinforcement fine-tuning workflow. The model receives prompts from your training dataset and generates several responses per prompt. These responses are then scored by a reward function. Amazon Bedrock uses the prompt-response pairs with scores to train the model through policy-based learning using Group Relative Policy Optimization (GRPO). The training loop continues until it reaches the end of your training data or you stop the job at a chosen checkpoint, producing a model optimized for the metric that matters to you. + +## Reinforcement fine-tuning best practices @@ -62 +63 @@ Amazon Bedrock fully automates the RFT workflow through a three-stage process: -**Stage 1: Response generation** + * **Start small** – Begin with 100-200 examples, validate reward function correctness, and scale gradually based on results @@ -64 +65 @@ Amazon Bedrock fully automates the RFT workflow through a three-stage process: -The actor model (the model being customized) receives prompts from your training dataset and generates responses. By default, it generates 4 responses per prompt. This stage supports both single-turn and multi-turn interactions, allowing comprehensive coverage of different use cases. + * **Pre fine-tuning evaluation** – Test baseline model performance before reinforcement fine-tuning. If rewards are consistently 0 percent, use supervised fine-tuning first to establish basic capabilities. If rewards are greater than 95 percent, reinforcement fine-tuning might be unnecessary @@ -66 +67 @@ The actor model (the model being customized) receives prompts from your training -**Stage 2: Reward computation** + * **Monitor training** – Track average reward scores and distribution. Watch for overfitting (training rewards increase while validation rewards decrease). Look for concerning patterns such as rewards plateauing below 0.15, increasing reward variance over time, and declining validation performance @@ -68 +69 @@ The actor model (the model being customized) receives prompts from your training -Actor model generated prompt-response pairs are evaluated by your selected optimizing models: + * **Optimize reward functions** – Execute within seconds (not minutes), minimize external API calls, use efficient algorithms, implement proper error handling, and take advantage of Lambda's parallel scaling @@ -70 +71 @@ Actor model generated prompt-response pairs are evaluated by your selected optim - * **RLVR** \- Execute through Lambda to compute objective scores + * **Iteration strategy** – If rewards aren't improving, adjust reward function design, increase dataset diversity, add more representative examples, and verify reward signals are clear and consistent @@ -72 +72,0 @@ Actor model generated prompt-response pairs are evaluated by your selected optim - * **RLAIF** \- Evaluate responses based on criteria and principles you configure (the console converts these into Lambda functions automatically) @@ -75,0 +76,7 @@ Actor model generated prompt-response pairs are evaluated by your selected optim +###### Topics + + * [Fine-tune Amazon Nova models with reinforcement fine-tuning](./rft-nova-models.html) + + * [Fine-tune open-weight models using OpenAI-compatible APIs](./fine-tuning-openai-apis.html) + + * [Evaluate your RFT model](./rft-evaluate-model.html) @@ -77 +83,0 @@ Actor model generated prompt-response pairs are evaluated by your selected optim -**Stage 3: Actor model training** @@ -79 +84,0 @@ Actor model generated prompt-response pairs are evaluated by your selected optim -Amazon Bedrock uses the prompt-response pairs with scores to train the actor model through policy-based learning using Group Relative Policy Optimization (GRPO). The training loop continues iteratively until the model achieves desired performance metrics or meets pre-defined stopping criteria. @@ -81 +85,0 @@ Amazon Bedrock uses the prompt-response pairs with scores to train the actor mod -Amazon Bedrock automatically handles parallel reward computation, training pipeline optimization, and implements safeguards against common reinforcement learning challenges like reward hacking and policy collapse. @@ -89 +93 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Submit a model fine-tuning or continued pre-training job +Submit a model fine-tuning job @@ -91 +95 @@ Submit a model fine-tuning or continued pre-training job -Reinforcement fine-tuning access and security +Fine-tune Amazon Nova models