AWS sagemaker documentation change
Summary
Expanded documentation for Direct Preference Optimization (DPO) with detailed data format requirements, configuration parameters, recipe examples for full-rank and LoRA DPO implementations, and added limitations section
Security assessment
The changes document DPO's ability to implement safety guardrails through preferred response patterns and improve model alignment for harmlessness. While this enhances safety capabilities, there's no evidence of addressing a specific security vulnerability. The documentation adds security-adjacent features but doesn't reference patching existing vulnerabilities.
Diff
diff --git a/sagemaker/latest/dg/nova-dpo.md b/sagemaker/latest/dg/nova-dpo.md index 86df24337..474f8dbfe 100644 --- a//sagemaker/latest/dg/nova-dpo.md +++ b//sagemaker/latest/dg/nova-dpo.md @@ -5 +5,437 @@ -# Direct Preference Optimization (DPO) +Full-rank DPOLow-rank adapter DPO + +# Direct preference optimization (DPO) + +Direct preference optimization (DPO) is an efficient fine-tuning method for foundation models that uses paired comparison data to align model outputs with human preferences. This approach enables direct optimization of model behavior based on human feedback about which responses are more desirable. + +Both full-rank DPO and low-rank adapter (LoRA) DPO are available. + +###### Data format requirements + +For both full-rank and LoRA DPO, the training data format requirements are similar to SFT. However, for DPO, the final turn needs to have preference pairs. Here is an example of the DPO data format: + + + // N-1 turns same as SFT format + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } // content list can contain multiple 'text' objects + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } // content list can contain multiple 'text' objects + ], + "preferenceLabel": "non-preferred" + } + ] + } + +Here is another complete DPO text sample: + + + { + "system": [ + { + "text": "..." + } + ], + "messages":[ + { + "role": "user", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "user", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "non-preferred" + } + ] + } + ], + } + +Here is a complete DPO image sample: + + + { + "system": [ + { + "text": "..." + } + ], + "messages":[ + { + "role": "user", + "content": [ + { + "text": "..." + }, + { + "text": "..." + }, + { + "image": { + "format": "jpeg", + "source": { + "s3Location": { + "uri": "s3://your-bucket/your-path/your-image.jpg", + "bucketOwner": "your-aws-account-id" + } + } + } + } // "content" can have multiple "text" and "image" objects. + // max image count is 10 + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "user", + "content": [ + { + "text": "..." + }, + { + "text": "..." + }, + { + "image": { + "format": "jpeg", + "source": { + "s3Location": { + "uri": "s3://your-bucket/your-path/your-image.jpg", + "bucketOwner": "your-aws-account-id" + } + } + } + } // "content" can have multiple "text" and "image" objects. + // max image count is 10 + ] + }, + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "non-preferred" + } + ] + } + ], + } + +Other constraints on the input datasets apply. For more information, see [Dataset constraints](https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-prepare-data-understanding.html#custom-fine-tune-constraints). We recommend that you include a minimum of 1,000 preference pairs for effective training. High-quality preference data leads to more efficient results. + +We recommend using DPO in the following scenarios: + + * Optimizing for subjective outputs that require alignment with specific human preferences. + + * Adjusting the model’s tone, style, or content characteristics to match desired response patterns. + + * Making targeted improvements to an existing model based on user feedback and error analysis. + + * Maintaining consistent output quality across different use cases. +