AWS aws-certification documentation change
Summary
Reformatted skills from heading style to bullet points and added new skills: 5.1.7 (agent performance frameworks), 5.1.8 (reporting systems), 5.1.9 (deployment validation), 5.2.2 (FM integration troubleshooting), 5.2.3 (prompt engineering troubleshooting), and 5.2.4 (retrieval system troubleshooting). Removed duplicate skill listings.
Security assessment
Changes are structural/organizational with no mention of security vulnerabilities, incidents, or protective measures. New skills focus on performance evaluation, troubleshooting, and reporting without security context.
Diff
diff --git a/aws-certification/latest/examguides/ai-professional-01-domain5.md b/aws-certification/latest/examguides/ai-professional-01-domain5.md index eddf6d01d..052f6e91d 100644 --- a//aws-certification/latest/examguides/ai-professional-01-domain5.md +++ b//aws-certification/latest/examguides/ai-professional-01-domain5.md @@ -11 +11 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.1: Develop comprehensive assessment frameworks to evaluate the quality and effectiveness of FM outputs beyond traditional ML evaluation approaches (for example, by using metrics for relevance, factual accuracy, consistency, and fluency). + * Skill 5.1.1: Develop comprehensive assessment frameworks to evaluate the quality and effectiveness of FM outputs beyond traditional ML evaluation approaches (for example, by using metrics for relevance, factual accuracy, consistency, and fluency). @@ -13 +13 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.2: Create systematic model evaluation systems to identify optimal configurations (for example, by using Amazon Bedrock Model Evaluations, A/B testing and canary testing of FMs, multi-model evaluation, cost-performance analysis to measure token efficiency, latency-to-quality ratios, and business outcomes). + * Skill 5.1.2: Create systematic model evaluation systems to identify optimal configurations (for example, by using Amazon Bedrock Model Evaluations, A/B testing and canary testing of FMs, multi-model evaluation, cost-performance analysis to measure token efficiency, latency-to-quality ratios, and business outcomes). @@ -15 +15 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.3: Develop user-centered evaluation mechanisms to continuously improve FM performance based on user experience (for example, by using feedback interfaces, rating systems for model outputs, annotation workflows to assess response quality). + * Skill 5.1.3: Develop user-centered evaluation mechanisms to continuously improve FM performance based on user experience (for example, by using feedback interfaces, rating systems for model outputs, annotation workflows to assess response quality). @@ -17 +17 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.4: Create systematic quality assurance processes to maintain consistent performance standards for FMs (for example, by using continuous evaluation workflows, regression testing for model outputs, automated quality gates for deployments). + * Skill 5.1.4: Create systematic quality assurance processes to maintain consistent performance standards for FMs (for example, by using continuous evaluation workflows, regression testing for model outputs, automated quality gates for deployments). @@ -19 +19 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.5: Develop comprehensive assessment systems to ensure thorough evaluation from multiple perspectives for FM outputs (for example, by using RAG evaluation, automated quality assessment with LLM-as-a-Judge techniques, human feedback collection interfaces). + * Skill 5.1.5: Develop comprehensive assessment systems to ensure thorough evaluation from multiple perspectives for FM outputs (for example, by using RAG evaluation, automated quality assessment with LLM-as-a-Judge techniques, human feedback collection interfaces). @@ -21 +21,7 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.6: Implement retrieval quality testing to evaluate and optimize information retrieval components for FM augmentation (for example, by using relevance scoring, context matching verification, retrieval latency measurements). + * Skill 5.1.6: Implement retrieval quality testing to evaluate and optimize information retrieval components for FM augmentation (for example, by using relevance scoring, context matching verification, retrieval latency measurements). + + * Skill 5.1.7: Develop agent performance frameworks to ensure that agents perform tasks correctly and efficiently (for example, by using task completion rate measurements, tool usage effectiveness evaluations, Amazon Bedrock Agent evaluations, reasoning quality assessment in multi-step workflows). + + * Skill 5.1.8: Create comprehensive reporting systems to communicate performance metrics and insights effectively to stakeholders for FM implementations (for example, by using visualization tools, automated reporting mechanisms, model comparison visualizations). + + * Skill 5.1.9: Create deployment validation systems to maintain reliability during FM updates (for example, by using synthetic user workflows, AI-specific output validation for hallucination rates and semantic drift, automated quality checks to ensure response consistency). @@ -23 +28,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.7: Develop agent performance frameworks to ensure that agents perform tasks correctly and efficiently (for example, by using task completion rate measurements, tool usage effectiveness evaluations, Amazon Bedrock Agent evaluations, reasoning quality assessment in multi-step workflows). @@ -25 +29,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.8: Create comprehensive reporting systems to communicate performance metrics and insights effectively to stakeholders for FM implementations (for example, by using visualization tools, automated reporting mechanisms, model comparison visualizations). @@ -27 +30,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.1.9: Create deployment validation systems to maintain reliability during FM updates (for example, by using synthetic user workflows, AI-specific output validation for hallucination rates and semantic drift, automated quality checks to ensure response consistency). @@ -31 +34,7 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.2.1: Resolve content handling issues to ensure that necessary information is processed completely in FM interactions (for example, by using context window overflow diagnostics, dynamic chunking strategies, prompt design optimization, truncation-related error analysis). + * Skill 5.2.1: Resolve content handling issues to ensure that necessary information is processed completely in FM interactions (for example, by using context window overflow diagnostics, dynamic chunking strategies, prompt design optimization, truncation-related error analysis). + + * Skill 5.2.2: Diagnose and resolve FM integration issues to identify and fix API integration problems specific to GenAI services (for example, by using error logging, request validation, response analysis). + + * Skill 5.2.3: Troubleshoot prompt engineering problems to improve FM response quality and consistency beyond basic prompt adjustments (for example, by using prompt testing frameworks, version comparison, systematic refinement). + + * Skill 5.2.4: Troubleshoot retrieval system issues to identify and resolve problems that affect information retrieval effectiveness for FM augmentation (for example, by using model response relevance analysis, embedding quality diagnostics, drift monitoring, vectorization issue resolution, chunking and preprocessing remediation, vector search performance optimization). @@ -33 +42 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.2.2: Diagnose and resolve FM integration issues to identify and fix API integration problems specific to GenAI services (for example, by using error logging, request validation, response analysis). + * Skill 5.2.5: Troubleshoot prompt maintenance issues to continuously improve the performance of FM interactions (for example, by using template testing and CloudWatch Logs to diagnose prompt confusion, X-Ray to implement prompt observability pipelines, schema validation to detect format inconsistencies, systematic prompt refinement workflows). @@ -35 +43,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.2.3: Troubleshoot prompt engineering problems to improve FM response quality and consistency beyond basic prompt adjustments (for example, by using prompt testing frameworks, version comparison, systematic refinement). @@ -37 +44,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.2.4: Troubleshoot retrieval system issues to identify and resolve problems that affect information retrieval effectiveness for FM augmentation (for example, by using model response relevance analysis, embedding quality diagnostics, drift monitoring, vectorization issue resolution, chunking and preprocessing remediation, vector search performance optimization). @@ -39 +45,0 @@ Task 5.1: Implement evaluation systems for GenAI.Task 5.2: Troubleshoot GenAI ap -### Skill 5.2.5: Troubleshoot prompt maintenance issues to continuously improve the performance of FM interactions (for example, by using template testing and CloudWatch Logs to diagnose prompt confusion, X-Ray to implement prompt observability pipelines, schema validation to detect format inconsistencies, systematic prompt refinement workflows).