AWS wellarchitected documentation change
Summary
Updated phrasing about model failure likelihood, added AI policy guidance for performance validation, and restructured related content sections
Security assessment
Changes focus on performance monitoring and organizational governance rather than security controls. The added AI policy guidance emphasizes performance standards but does not introduce security features or address vulnerabilities.
Diff
diff --git a/wellarchitected/latest/generative-ai-lens/genops01-bp01.md b/wellarchitected/latest/generative-ai-lens/genops01-bp01.md index c2a37f00f..f451199f4 100644 --- a//wellarchitected/latest/generative-ai-lens/genops01-bp01.md +++ b//wellarchitected/latest/generative-ai-lens/genops01-bp01.md @@ -17 +17 @@ Implement periodic evaluations using stratified sampling and custom metrics to m - * [Anticipate failure](https://docs.aws.amazon.com/wellarchitected/latest/framework/oe-design-principles.html) \- Periodic review of the model's performance levels helps you proactively identify deviations in its performance. This is because foundation models are inherently non-deterministic, implying there is always a chance of failure. + * [Anticipate failure](https://docs.aws.amazon.com/wellarchitected/latest/framework/oe-design-principles.html) \- Periodic review of the model's performance levels helps you proactively identify deviations in its performance. This is because foundation models are inherently non-deterministic with a realistic chance of failure. @@ -30 +30 @@ You can employ stratified sampling techniques to verify diverse data representat -You can use the model evaluation feature built-in with Amazon Bedrock or open-source libraries like [fmeval](https://github.com/aws/fmeval) or [ragas](https://docs.ragas.io/en/stable/). Use Amazon Bedrock model invocation logging to collect metadata, requests, and responses for all model invocations in your account. +You can use the model evaluation feature built-in with Amazon Bedrock or open-source libraries like [fmeval](https://github.com/aws/fmeval) or [ragas](https://docs.ragas.io/en/stable/). Use Amazon Bedrock model invocation logging to collect metadata, requests, and responses for model invocations in your account. @@ -35,0 +36,2 @@ The fmeval library provides a framework for defining and using custom metrics. B +Your organization’s AI policy should define the effective minimum performance levels for generative AI workloads, as well as how to validate performance on an ongoing basis. Consider identifying a single-threaded workload owner responsible for the operational considerations pertaining to ongoing performance evaluations. Run these evaluations when new candidate models are available, or when model customization techniques are applied. For example, fine-tuned and customized models should be subject to the same evaluation criteria and cadence as non-customized models. + @@ -112 +114 @@ The fmeval library provides a framework for defining and using custom metrics. B -**Related practices:** +**Related best practices:** @@ -119 +121 @@ The fmeval library provides a framework for defining and using custom metrics. B -**Related guides, videos, and documentation:** +**Related documents:** @@ -126,0 +129,5 @@ The fmeval library provides a framework for defining and using custom metrics. B + + + +**Related videos:** +