AWS nova documentation change
Summary
Updated reinforcement fine-tuning documentation with configuration changes including added generation_replicas parameter, updated rollout strategy with off_policy_async type, increased number_generation from 2 to 4, changed kl_loss_coef from 0.001 to 0.0, reduced learning rate from 5e-5 to 1e-6, and various clarifications and formatting improvements.
Security assessment
This change contains routine documentation updates for machine learning configuration parameters without any mention of security vulnerabilities, patches, or security incidents. The changes focus on improving clarity, adding configuration options, and updating parameter values for reinforcement fine-tuning workflows. There is no evidence of addressing security weaknesses or adding security features.
Diff
diff --git a/nova/latest/nova2-userguide/nova-reinforcement-fine-tuning.md b/nova/latest/nova2-userguide/nova-reinforcement-fine-tuning.md index 0dea934eb..6f62c7afa 100644 --- a//nova/latest/nova2-userguide/nova-reinforcement-fine-tuning.md +++ b//nova/latest/nova2-userguide/nova-reinforcement-fine-tuning.md @@ -15 +15 @@ OverviewData format overviewDataset size recommendationsCharacteristics of effec -Reinforcement fine-tuning (RFT) improves model performance by training on feedback signals—measurable scores or rewards indicating how well the model performed—rather than exact correct answers. Unlike supervised fine-tuning that learns from input-output pairs, RFT uses reward functions to evaluate model responses and iteratively optimizes the model to maximize these rewards. This approach excels when defining the exact correct output is challenging, but you can reliably measure response quality. +Reinforcement fine-tuning (RFT) improves model performance by training on feedback signals—measurable scores or rewards indicating how well the model performed—rather than exact correct answers. Unlike supervised fine-tuning (SFT) that learns from input-output pairs, RFT uses reward functions to evaluate model responses and iteratively optimizes the model to maximize these rewards. This approach excels when defining the exact correct output is challenging, but you can reliably measure response quality. @@ -19 +19 @@ Reinforcement fine-tuning (RFT) improves model performance by training on feedba -Use RFT when you can define clear, measurable success criteria but struggle to provide exact correct outputs for training. It's ideal for: +Use RFT when you can define clear, measurable success criteria but it's difficult to provide exact correct outputs for training. RFT is ideal for: @@ -307 +307 @@ These additional fields are passed to your reward function during evaluation, en - # This recipe can run on p5.48xlarge and p5en.48xlarge instance types. + # This recipe can run on p5.48xlarge, p5e.48xlarge, and p5en.48xlarge instance types. @@ -309 +309 @@ These additional fields are passed to your reward function during evaluation, en - name: "my-rft-run" # Unique run name (appears in logs/artifacts). + name: "my-rft-run" # Unique run name (appears in logs and artifacts). @@ -312,2 +312,3 @@ These additional fields are passed to your reward function during evaluation, en - data_s3_path: s3://<bucket>/<data file> # Training dataset in JSONL; - replicas: 4 + data_s3_path: s3://<bucket>/<data-file> # Training dataset in JSONL format. + replicas: 4 # Number of total training instances. + generation_replicas: 2 # Number of total instances dedicated to response generation. @@ -321 +322 @@ These additional fields are passed to your reward function during evaluation, en - ## SMTJ GRPO Training specific configs + ## SMTJ RFT training configs @@ -323,3 +324,3 @@ These additional fields are passed to your reward function during evaluation, en - max_length: 8192 # Context window (tokens) for inputs+prompt; - global_batch_size: 16 # Total samples per optimizer step across all replicas (16/32/64/128/256). - reasoning_effort: high # Enables reasoning mode high / low / or null for non-reasoning + max_length: 8192 # Context window (tokens) for inputs and prompt. + global_batch_size: 32 # Total samples per optimizer step across all replicas (16/32/64/128/256). + reasoning_effort: high # Reasoning mode: high, low, or null for non-reasoning. @@ -327 +328,7 @@ These additional fields are passed to your reward function during evaluation, en - rollout: # How responses are generated for GRPO/advantage calc. + data: + shuffle: true # Shuffle training data each epoch. + + rollout: # Controls how responses are generated for advantage calculation. + rollout_strategy: + type: off_policy_async # Asynchronous rollout for higher throughput. + age_tolerance: 2 # Maximum policy age before regeneration. @@ -329 +336 @@ These additional fields are passed to your reward function during evaluation, en - number_generation: 2 # N samples per prompt to estimate advantages (variance vs cost). + number_generation: 4 # Samples per prompt to estimate advantages (higher = lower variance but higher cost). @@ -331 +338 @@ These additional fields are passed to your reward function during evaluation, en - max_new_tokens: 6000 # Cap on tokens generated per sample + max_new_tokens: 6000 # Cap on tokens generated per sample. @@ -333,2 +340,2 @@ These additional fields are passed to your reward function during evaluation, en - temperature: 1 # Softmax temperature; - top_k: 1 # Sample only from top-K logits + temperature: 1 # Softmax temperature for sampling. + top_k: 1 # Sample only from top-K logits. @@ -336 +343 @@ These additional fields are passed to your reward function during evaluation, en - preset_reward_function: null # Usage of preset reward functions [exact_match] + preset_reward_function: null # Preset reward functions: exact_match or null for custom. @@ -339,0 +347 @@ These additional fields are passed to your reward function during evaluation, en + lambda_batch_size: 128 # Number of samples per Lambda invocation. @@ -342,3 +350,6 @@ These additional fields are passed to your reward function during evaluation, en - max_steps: 2 # Steps to train for. One Step = global_batch_size - save_steps: 5 - test_steps: 1 + max_steps: 2 # Steps to train for. One step = global_batch_size samples. + save_steps: 5 # Save a checkpoint every N steps. + test_steps: 1 # Run validation every N reference model updates. + refit_freq: 4 # Frequency of reference model updates. + clip_ratio_high: 0.2 # PPO clip ratio for policy updates. + loss_scale: 1.0 # Scaling factor for the policy loss. @@ -347,2 +358,2 @@ These additional fields are passed to your reward function during evaluation, en - ent_coeff: 0.0 # A bonus added to the policy loss that rewards higher-output entropy. - kl_loss_coef: 0.001 # Weight on the KL penalty between the actor (trainable policy) and a frozen reference model + ent_coeff: 0.0 # Entropy bonus added to the policy loss (higher = more exploration). + kl_loss_coef: 0.0 # Weight on the KL penalty between the current and reference policy. @@ -350,3 +361,3 @@ These additional fields are passed to your reward function during evaluation, en - optim_config: # Optimizer settings - lr: 5e-5 # Learning rate - weight_decay: 0.0 # L2 regularization strength (0.0–1.0) + optim_config: # Optimizer settings. + lr: 1e-6 # Learning rate. + weight_decay: 0.0 # L2 regularization strength (0.0 to 1.0). @@ -356,2 +367,2 @@ These additional fields are passed to your reward function during evaluation, en - peft: # Parameter-efficient fine-tuning (LoRA) - peft_scheme: "lora" # Enable LoRA for PEFT + peft: # Parameter-efficient fine-tuning (LoRA). + peft_scheme: "lora" # Enable LoRA for PEFT. @@ -359,2 +370,2 @@ These additional fields are passed to your reward function during evaluation, en - alpha: 32 - lora_plus_lr_ratio: 64.0 # LoRA+ learning rate scaling factor (0.0–100.0) + alpha: 64 # LoRA scaling factor. + lora_plus_lr_ratio: 64.0 # LoRA+ learning rate scaling factor (0.0 to 100.0).