AWS wellarchitected documentation change
Summary
Expanded guidance on ground truth data for generative AI, including metadata requirements, organizational policy alignment, testing automation, and model customization workflows. Added implementation steps for data storage, dictionary creation, and policy-compliant testing.
Security assessment
The changes emphasize developing ground truth data in accordance with organizational AI policies (including prohibitions against testing with production data) and implementing mock datasets. While this demonstrates security-conscious data handling practices, there is no explicit mention of addressing vulnerabilities or security incidents.
Diff
diff --git a/wellarchitected/latest/generative-ai-lens/genperf01-bp01.md b/wellarchitected/latest/generative-ai-lens/genperf01-bp01.md index 9ed656f0f..89ab5b0ba 100644 --- a//wellarchitected/latest/generative-ai-lens/genperf01-bp01.md +++ b//wellarchitected/latest/generative-ai-lens/genperf01-bp01.md @@ -9 +9 @@ Implementation guidanceResources -_Ground truth data_ facilitates model testing for use case specific scenarios and should be developed and curated for generative AI workloads. +_Ground truth data_ facilitates model testing for use case specific scenarios and should be developed and curated for generative AI workloads. Ground truth data is a curated set of prompts and responses that describe the ideal workflow with a model. @@ -11 +11 @@ _Ground truth data_ facilitates model testing for use case specific scenarios an -**Desired outcome:** When implemented, this best practice improves the performance of model selection by measuring a model's performance on task specific prompt-response pairs. +**Desired outcome:** When implemented, this best practice enables the measurement of a model's performance for a set of tasks, accelerating model evaluation and enabling model customization workflows. @@ -13 +13 @@ _Ground truth data_ facilitates model testing for use case specific scenarios an -**Benefits of establishing this best practice:** [Experiment more often](https://docs.aws.amazon.com/wellarchitected/latest/framework/rel-dp.html) \- Ground truth testing facilitates rapid experimentation for models on tasks specific to your workload's unique requirements. +**Benefits of establishing this best practice:** [Experiment more often](https://docs.aws.amazon.com/wellarchitected/latest/framework/rel-dp.html) \- Ground truth testing facilitates rapid experimentation and customization for models on tasks specific to your workload's unique requirements. @@ -19 +19 @@ _Ground truth data_ facilitates model testing for use case specific scenarios an -_Ground truth data_ , also known as a _golden dataset_ , is data classified to be true. Ground truth data is vital for the efficient testing of data-driven workloads, particularly generative AI workloads. Customers should develop ground truth data for their generative AI applications to facilitate the testing process. +_Ground truth data_ , also known as a _golden dataset_ , is data considered to be of the highest quality in regard to a specific use case. Ground truth data for generative AI workloads are oftentimes prompt-response pairs. For a simple workflow, a golden dataset might be dozens, hundreds, thousands or more sample prompts and their corresponding expected responses. There may be several prompts containing variations of the same ask, with several responses describing variations of an acceptable response. More complex workflows like retrieval augmented generation or agentic workflows may require variations on this paradigm. @@ -21 +21 @@ _Ground truth data_ , also known as a _golden dataset_ , is data classified to b -Ground truth data for generative AI traditionally consists of a prompt and a desirable response to said prompt. For prompts that supplement responses with data from external sources, customers can extend the ground truth data to include source documentation or other useful metadata. At a minimum, a prompt and a sufficiently acceptable response are required for a usable ground truth data set. +Ground truth data is vital for the efficient testing of data-driven and generative AI workloads. Develop ground truth data for your generative AI applications to facilitate the rigorous and uniform testing of large language models. When equipped with a ground truth dataset for a use case, you can automate the testing and evaluation of models. New models can quickly be evaluated to determine if their performance for a specific use case meets the current model's high bar. @@ -23 +23,7 @@ Ground truth data for generative AI traditionally consists of a prompt and a des -Ground truth data should be considered a living artifact, one that changes and extends based on the use cases being tested. For generative AI workloads, ground truth prompts should be clear and succinct, but not repeated. Ground truth responses should similarly be clear and succinct, and responses can be repeated if a response addresses multiple prompts. When developing a ground truth data set, don't be overly concerned with slight differences in prompts that essentially ask a model to perform the same task. Prompts in the ground truth data set should be specific to the kinds of tasks you expect a model to solve. +Ground truth prompts should be clear and succinct, grouped together by variations of the same ask. Ground truth responses should similarly be clear and succinct, covering a range of acceptable responses. When developing a ground truth data set, don't be overly concerned with slight differences in prompts that essentially ask a model to perform the same task. Prompts in the ground truth data set should be specific to the kinds of tasks you expect a model to solve. Consider ground truth data as a living artifact, one that changes and extends based on the use cases being tested and the usage paradigms being implemented. + +Prompt-responses pairs are the core of a ground truth dataset, but ground truth data needs additional meta-data to be viable for the extent of generative AI usage paradigms that could be tested. For example, agent workflows perform tasks on behalf of a requester, using its judgment to discern how to interpret a response from an external system. An agent workflow may synthesize several intermediary responses before the language model delivers a final response to the user. Ground truth data should be able to capture an ideal prompt flow, tracing the workflow of the agent through various systems. This same practice could be applied to workflows interacting with multiple models. + +Develop ground truth data in accordance with your organization's AI policy. For example, if your organization's AI policy prohibits testing models against production data, your golden dataset should contain references to data which is functionally equivalent to production data. Develop mock data sets for testing, and mock endpoints for testing agentic flows. The golden dataset should contain the instructions required for a testing harness to run tests autonomously against any model endpoint available, including self-hosted language models. + +In addition to facilitating rapid model testing and evaluation, golden datasets can be used to quickly fine-tune models or distill student models from teacher models. Model customization workflows require high-quality data for customization. Maintaining a robust golden dataset for each use case can accelerate your ability to customize models. @@ -27 +33,5 @@ Ground truth data should be considered a living artifact, one that changes and e - 1. Define a series of prompts and their expected responses. Consider using Amazon SageMaker Ground Truth or similar to scale the curation of this dataset. + 1. Define a series of prompts and their expected responses. + + * Consider using Amazon SageMaker Ground Truth or similar to scale the curation of this dataset. + + * Enrich prompt-response pairs with relevant meta-data in accordance with your organization's AI policy. @@ -29 +39 @@ Ground truth data should be considered a living artifact, one that changes and e - 2. Create a nested dictionary of data. + 2. Store the data in a way which facilitates a dictionary-style lookup of the data. @@ -31 +41 @@ Ground truth data should be considered a living artifact, one that changes and e - * The first several layers are organizational, referring to abstractions like language, business domain, or use case. + * The first several layers could be organizational, referring to abstractions like language, business domain, or use case. @@ -35 +45 @@ Ground truth data should be considered a living artifact, one that changes and e - * Store the dictionary in object-storage or a database. + * Store the data in an object-store such as Amazon S3. @@ -37 +47,3 @@ Ground truth data should be considered a living artifact, one that changes and e - 3. Define test scenarios corresponding to your golden dataset. + 3. Create a data dictionary to facilitate access to the ground truth data. + + * Crawl the object-store using an AWS Glue Crawler to build the data dictionary. @@ -40,0 +53,10 @@ Ground truth data should be considered a living artifact, one that changes and e + * Query segments of the ground-truth dataset using a federated query solution such as Amazon Athena. + + * Incorporate mock production data and tooling for more advanced workflows such as agents or RAG. + + 5. Define test scenarios corresponding to your golden dataset and adhere to your organization's AI policy. + + * Define metrics to test models against as may be required by your organization's AI policy. + + * Track model performance across various tests and metrics, carefully evaluating the trade-offs across models. + @@ -46 +68 @@ Ground truth data should be considered a living artifact, one that changes and e -**Related practices:** +**Related best practices:** @@ -51,0 +74,5 @@ Ground truth data should be considered a living artifact, one that changes and e + * [MLPER03](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/mlper-03.html) + + * [MLPER04](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/mlper-04.html) + + * [MLPER16](https://docs.aws.amazon.com/wellarchitected/latest/machine-learning-lens/mlper-16.html) @@ -55 +82,6 @@ Ground truth data should be considered a living artifact, one that changes and e -**Related guides, videos, and documentation:** + +**Related documents:** + + * [Understand options for evaluating large language models with SageMaker AI Clarify](https://docs.aws.amazon.com/sagemaker/latest/dg/clarify-foundation-model-evaluate.html) + + * [Customize your model to improve its performance for your use case](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html) @@ -65,0 +98,2 @@ Ground truth data should be considered a living artifact, one that changes and e + * [Customize models in Amazon Bedrock with your own data using fine-tuning and continued pre-training](https://aws.amazon.com/blogs/aws/customize-models-in-amazon-bedrock-with-your-own-data-using-fine-tuning-and-continued-pre-training/) +