AWS nova documentation change
Summary
Removed detailed DPO configuration documentation and replaced with a link to SageMaker DPO guide. Deleted sections include data format requirements, parameter configurations, example recipes, and limitations.
Security assessment
The changes remove technical implementation details but don't address security vulnerabilities or introduce security features. The mention of 'reduce harmful outputs' in the updated DPO description relates to general model alignment rather than specific security controls.
Diff
diff --git a/nova/latest/userguide/customize-fine-tune-hyperpod-dpo.md b/nova/latest/userguide/customize-fine-tune-hyperpod-dpo.md index 91340a37b..c4332e540 100644 --- a//nova/latest/userguide/customize-fine-tune-hyperpod-dpo.md +++ b//nova/latest/userguide/customize-fine-tune-hyperpod-dpo.md @@ -5,2 +4,0 @@ -Full-rank DPOLow-rank adapter DPO - @@ -9 +7 @@ Full-rank DPOLow-rank adapter DPO -Direct preference optimization (DPO) is an efficient fine-tuning method for foundation models that uses paired comparison data to align model outputs with human preferences. This approach enables direct optimization of model behavior based on human feedback about which responses are more desirable. +DPO is an advanced technique that fine-tunes models based on human preferences rather than fixed labels. It uses paired examples where humans have indicated which response is better for a given prompt. The model learns to generate outputs that align with these preferences, helping to improve response quality, reduce harmful outputs, and better align with human values. DPO is particularly valuable for refining model behavior after initial SFT. @@ -13,432 +11 @@ Both full-rank DPO and low-rank adapter (LoRA) DPO are available. -###### Data format requirements - -For both full-rank and LoRA DPO, the training data format requirements are similar to SFT. However, for DPO, the final turn needs to have preference pairs. Here is an example of the DPO data format: - - - // N-1 turns same as SFT format - { - "role": "assistant", - "candidates": [ - { - "content": [ - { - "text": "..." - } // content list can contain multiple 'text' objects - ], - "preferenceLabel": "preferred" - }, - { - "content": [ - { - "text": "..." - } // content list can contain multiple 'text' objects - ], - "preferenceLabel": "non-preferred" - } - ] - } - -Here is another complete DPO text sample: - - - { - "system": [ - { - "text": "..." - } - ], - "messages":[ - { - "role": "user", - "content": [ - { - "text": "..." - } - ] - }, - { - "role": "assistant", - "content": [ - { - "text": "..." - } - ] - }, - { - "role": "user", - "content": [ - { - "text": "..." - } - ] - }, - { - "role": "assistant", - "candidates": [ - { - "content": [ - { - "text": "..." - } - ], - "preferenceLabel": "preferred" - }, - { - "content": [ - { - "text": "..." - } - ], - "preferenceLabel": "non-preferred" - } - ] - } - ], - } - -Here is a complete DPO image sample: - - - { - "system": [ - { - "text": "..." - } - ], - "messages":[ - { - "role": "user", - "content": [ - { - "text": "..." - }, - { - "text": "..." - }, - { - "image": { - "format": "jpeg", - "source": { - "s3Location": { - "uri": "s3://your-bucket/your-path/your-image.jpg", - "bucketOwner": "your-aws-account-id" - } - } - } - } // "content" can have multiple "text" and "image" objects. - // max image count is 10 - ] - }, - { - "role": "assistant", - "content": [ - { - "text": "..." - } - ] - }, - { - "role": "user", - "content": [ - { - "text": "..." - }, - { - "text": "..." - }, - { - "image": { - "format": "jpeg", - "source": { - "s3Location": { - "uri": "s3://your-bucket/your-path/your-image.jpg", - "bucketOwner": "your-aws-account-id" - } - } - } - } // "content" can have multiple "text" and "image" objects. - // max image count is 10 - ] - }, - { - "role": "assistant", - "candidates": [ - { - "content": [ - { - "text": "..." - } - ], - "preferenceLabel": "preferred" - }, - { - "content": [ - { - "text": "..." - } - ], - "preferenceLabel": "non-preferred" - } - ] - } - ], - } - -Other constraints on the input datasets apply. For more information, see [Dataset constraints](https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-prepare-data-understanding.html#custom-fine-tune-constraints). We recommend that you include a minimum of 1,000 preference pairs for effective training. High-quality preference data leads to more efficient results. - -We recommend using DPO in the following scenarios: - - * Optimizing for subjective outputs that require alignment with specific human preferences. - - * Adjusting the model’s tone, style, or content characteristics to match desired response patterns. - - * Making targeted improvements to an existing model based on user feedback and error analysis. - - * Maintaining consistent output quality across different use cases. - - * Implementing safety guardrails through preferred response patterns. - - * Training with reward-free reinforcement learning.