AWS wellarchitected documentation change
Summary
Complete overhaul of documentation: Changed topic from performance requirement evaluation to generative AI component selection. Replaced all best practices with new AI-focused recommendations including ground truth datasets, model customization, and vector store optimization.
Security assessment
The changes focus entirely on generative AI implementation without addressing security vulnerabilities. However, BP01 adds security documentation by recommending secure storage with access controls for sensitive financial datasets. No evidence of addressing specific security incidents.
Diff
diff --git a/wellarchitected/latest/financial-services-industry-lens/fsiperf05.md b/wellarchitected/latest/financial-services-industry-lens/fsiperf05.md index f3ea16a71..7f1c77cce 100644 --- a//wellarchitected/latest/financial-services-industry-lens/fsiperf05.md +++ b//wellarchitected/latest/financial-services-industry-lens/fsiperf05.md @@ -5 +5 @@ -FSIPERF05-BP01 Use Application Performance Monitoring (APM) toolsFSIPERF05-BP02 Integrate performance testing into the release cycleFSIPERF05-BP03 Verify consistency and failure recovery during load testsFSIPERF05-BP04 Understand performance of the system under peak load and in failure scenariosFSIPERF05-BP05 Include dependencies in your load tests +FSIPERF05-BP01 Define a ground truth data set of prompts and responses for financial services use casesFSIPERF05-BP02 Select and customize models appropriate for financial services use casesFSIPERF05-BP03 Optimize vector stores for financial data retrieval @@ -7 +7 @@ FSIPERF05-BP01 Use Application Performance Monitoring (APM) toolsFSIPERF05-BP02 -# FSIPERF05: How do you evaluate compliance with performance requirements? +# FSIPERF05: How do you select and optimize generative AI components for your workload? @@ -9 +9 @@ FSIPERF05-BP01 Use Application Performance Monitoring (APM) toolsFSIPERF05-BP02 -Here are several methods for doing so: +Selecting and optimizing generative AI components requires defining your use case requirements—including accuracy thresholds, latency constraints, and compliance needs—then evaluating foundation models using automated benchmarks and task-specific criteria before optimizing through prompt engineering or fine-tuning. This enables you to build generative AI systems that deliver reliable business value while meeting the rigorous standards required for production deployment, particularly in regulated industries like financial services. @@ -11 +11 @@ Here are several methods for doing so: - * Monitoring of your workload at multiple levels helps verify that your resources are performing as expected and you are aware of deviations. +## FSIPERF05-BP01 Define a ground truth data set of prompts and responses for financial services use cases @@ -13 +13 @@ Here are several methods for doing so: - * Consider all dimensions of the solution for monitoring, for example client-side and server-side metrics, application metrics and infrastructure metrics, technical and functional metrics. +For financial services applications using generative AI, develop a ground truth dataset that captures domain-specific prompts and expected responses. This dataset should include scenarios relevant to financial applications such as regulatory adherence queries, transaction anomaly detection, risk assessment, and customer service interactions common in financial institutions. @@ -15 +15 @@ Here are several methods for doing so: - * Monitor for failure rates and alert when they are above expected values. +**Implementation steps:** @@ -17 +17 @@ Here are several methods for doing so: - * Identify KPIs and create threshold alerts for them and determine what actions to take (like autoscaling) when thresholds are breached - this allows you to observe the overall health of your system and identify [non-binary, or grey, failure states](https://docs.aws.amazon.com/whitepapers/latest/advanced-multi-az-resilience-patterns/gray-failures.html). + * Define a series of prompts and expected responses specific to financial services use cases. @@ -19 +19 @@ Here are several methods for doing so: - * Provide visibility of data loss in your metrics, for example by monitoring for lost messages. + * Create a structured dataset that organizes these prompt-response pairs by business domain (like banking, trading, and risk management). @@ -21 +21 @@ Here are several methods for doing so: - * Where possible capture inter-solution and inter-process communication streams to aid with the reproduction of issues. + * Store this dataset in a secure object storage or database with appropriate access controls given the sensitive nature of financial data. @@ -22,0 +23 @@ Here are several methods for doing so: + * Develop a testing harness that can evaluate model performance against these financial services scenarios. @@ -26 +26,0 @@ Here are several methods for doing so: -## FSIPERF05-BP01 Use Application Performance Monitoring (APM) tools @@ -28 +28 @@ Here are several methods for doing so: -Use APM tools to provide your organization the capability to verify that application performance meets its defined requirements. AWS offers features and services to monitor and subsequently right-size the cloud services that you need to meet performance requirements. +## FSIPERF05-BP02 Select and customize models appropriate for financial services use cases @@ -30 +30 @@ Use APM tools to provide your organization the capability to verify that applica -For example, you can monitor and set alarms on latency and error rates for each user request using Amazon CloudWatch metrics and alarms, or on your downstream dependencies, or on the success and failure of key operations. Amazon CloudWatch Synthetics can be used to create _canaries_ , configurable scripts that run on a schedule, or to monitor your endpoints, and APIs. +When implementing generative AI models in financial services workloads, evaluate model performance against domain-specific requirements including regulatory adherence, accuracy in financial terminology, and consistency in risk assessment. Consider model customization through fine-tuning or continuous pre-training to improve performance on financial domain knowledge and financial institution-specific scenarios. @@ -32 +32 @@ For example, you can monitor and set alarms on latency and error rates for each -The required level of monitoring generates huge amounts of data, which can be challenging for operation teams to store, analyze, and visualize, so make use of services including Amazon Managed Service for Prometheus to monitor and alert on containers, Amazon Managed Grafana to visualize metrics and logs, and the wide range of features found in Amazon CloudWatch, to provide the appropriate tools for monitoring your systems without the overhead of managing additional infrastructure. Teams need training to update their skills and processes and take full advantage of this new fidelity of insight. +**Implementation steps:** @@ -34 +34 @@ The required level of monitoring generates huge amounts of data, which can be ch -## FSIPERF05-BP02 Integrate performance testing into the release cycle + * Test multiple models against your financial services ground truth dataset. @@ -36 +36 @@ The required level of monitoring generates huge amounts of data, which can be ch -Rather than considering performance testing to be a separate part of the workload release cycle, integrate performance testing into your release process and CI/CD tooling. This allows you to record and evaluate performance metrics across releases, being aware of and taking action when metrics change as early as possible. + * Consider customizing models using techniques like fine-tuning to improve performance on financial tasks. @@ -38 +38 @@ Rather than considering performance testing to be a separate part of the workloa -## FSIPERF05-BP03 Verify consistency and failure recovery during load tests + * Evaluate model response consistency and accuracy, particularly for regulated processes. @@ -40 +40 @@ Rather than considering performance testing to be a separate part of the workloa -You must verify data consistency and recovery during periods of high load. Ensuring that your workload's RTO and RPO is still valid under the highest load can uncover gaps in your architecture and operational resilience. + * Consider model distillation techniques for deploying smaller, more efficient models in production that maintain accuracy for specific financial tasks. @@ -42 +41,0 @@ You must verify data consistency and recovery during periods of high load. Ensur -## FSIPERF05-BP04 Understand performance of the system under peak load and in failure scenarios @@ -44 +42,0 @@ You must verify data consistency and recovery during periods of high load. Ensur -Include testing of common failure scenarios in your performance testing suites to understand your workload behaviour in these situations and determine areas for improvement. @@ -46 +43,0 @@ Include testing of common failure scenarios in your performance testing suites t -Extend the range of performance testing scenarios to cover testing at loads beyond current peak loads, and testing the scaling processes themselves of the application to understand how the environment behaves under increasing load. @@ -48 +45,16 @@ Extend the range of performance testing scenarios to cover testing at loads beyo -Under common or anticipated failure scenarios, workloads should exhibit predictable failure patterns with performance degrading gracefully using techniques such as [fail-open behavior,](https://www.wellarchitectedlabs.com/reliability/300_labs/300_health_checks_and_dependencies/4_fail_open/) and the transformation of [hard dependencies into soft dependencies.](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_mitigate_interaction_failure_graceful_degradation.html) +## FSIPERF05-BP03 Optimize vector stores for financial data retrieval + +Financial services applications often require high-precision data retrieval from large datasets of financial information, regulatory documents, or transaction histories. Optimize vector databases to enhance the retrieval accuracy and speed when used in conjunction with generative AI models. + +**Implementation steps:** + + * Test different chunking strategies for financial documents, considering their specialized structure. + + * Select appropriate approximate nearest neighbor (ANN) algorithms based on the precision and recall requirements for financial use cases. + + * Optimize vector dimensions based on the complexity and specificity of financial information. + + * Implement hierarchical indices that allow efficient navigation from general financial concepts to specific details. + + * Regularly test and monitor performance metrics including latency, throughput, and accuracy. + @@ -50 +61,0 @@ Under common or anticipated failure scenarios, workloads should exhibit predicta -## FSIPERF05-BP05 Include dependencies in your load tests @@ -52 +62,0 @@ Under common or anticipated failure scenarios, workloads should exhibit predicta -Financial institutions need to map resources they need to continuously deliver their important business services. These resources are your people, processes, technology, facilities, and information, including third-party service providers. This mapping allows the identification of operational dependencies, vulnerabilities, and threats. Incorporating the dependencies of your workload (such as on financial messaging providers) as part of your performance tests enables you to demonstrate the overall resiliency of your workload. @@ -60 +70 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Monitoring +FSIPERF04: How do you select your network architecture? @@ -62 +72 @@ Monitoring -Trade-offs +Monitoring