AWS sagemaker documentation change
Summary
Minor grammatical correction changing 'HyperPod CLI' to 'the HyperPod CLI' in section header
Security assessment
Change is purely a grammatical improvement with no security implications or documentation of security features
Diff
diff --git a/sagemaker/latest/dg/nova-dpo.md b/sagemaker/latest/dg/nova-dpo.md index 86df24337..474f8dbfe 100644 --- a//sagemaker/latest/dg/nova-dpo.md +++ b//sagemaker/latest/dg/nova-dpo.md @@ -5 +5,437 @@ -# Direct Preference Optimization (DPO) +Full-rank DPOLow-rank adapter DPO + +# Direct preference optimization (DPO) + +Direct preference optimization (DPO) is an efficient fine-tuning method for foundation models that uses paired comparison data to align model outputs with human preferences. This approach enables direct optimization of model behavior based on human feedback about which responses are more desirable. + +Both full-rank DPO and low-rank adapter (LoRA) DPO are available. + +###### Data format requirements + +For both full-rank and LoRA DPO, the training data format requirements are similar to SFT. However, for DPO, the final turn needs to have preference pairs. Here is an example of the DPO data format: + + + // N-1 turns same as SFT format + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } // content list can contain multiple 'text' objects + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } // content list can contain multiple 'text' objects + ], + "preferenceLabel": "non-preferred" + } + ] + } + +Here is another complete DPO text sample: + + + { + "system": [ + { + "text": "..." + } + ], + "messages":[ + { + "role": "user", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "user", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "non-preferred" + } + ] + } + ], + } + +Here is a complete DPO image sample: + + + { + "system": [ + { + "text": "..." + } + ], + "messages":[ + { + "role": "user", + "content": [ + { + "text": "..." + }, + { + "text": "..." + }, + { + "image": { + "format": "jpeg", + "source": { + "s3Location": { + "uri": "s3://your-bucket/your-path/your-image.jpg", + "bucketOwner": "your-aws-account-id" + } + } + } + } // "content" can have multiple "text" and "image" objects. + // max image count is 10 + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "..." + } + ] + }, + { + "role": "user", + "content": [ + { + "text": "..." + }, + { + "text": "..." + }, + { + "image": { + "format": "jpeg", + "source": { + "s3Location": { + "uri": "s3://your-bucket/your-path/your-image.jpg", + "bucketOwner": "your-aws-account-id" + } + } + } + } // "content" can have multiple "text" and "image" objects. + // max image count is 10 + ] + }, + { + "role": "assistant", + "candidates": [ + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "preferred" + }, + { + "content": [ + { + "text": "..." + } + ], + "preferenceLabel": "non-preferred" + } + ] + } + ], + } + +Other constraints on the input datasets apply. For more information, see [Dataset constraints](https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-prepare-data-understanding.html#custom-fine-tune-constraints). We recommend that you include a minimum of 1,000 preference pairs for effective training. High-quality preference data leads to more efficient results. + +We recommend using DPO in the following scenarios: + + * Optimizing for subjective outputs that require alignment with specific human preferences. + + * Adjusting the model’s tone, style, or content characteristics to match desired response patterns. + + * Making targeted improvements to an existing model based on user feedback and error analysis. + + * Maintaining consistent output quality across different use cases. +