AWS bedrock documentation change
Summary
Added documentation for fidelity reports including console/API review steps, coverage/accuracy analysis, and collaboration tips
Security assessment
Changes focus on policy validation accuracy and document coverage metrics without addressing vulnerabilities, threats, or security controls. The fidelity report is a quality assurance feature for policy extraction accuracy.
Diff
diff --git a/bedrock/latest/userguide/create-automated-reasoning-policy.md b/bedrock/latest/userguide/create-automated-reasoning-policy.md index 064d044b6..7d88ddebf 100644 --- a//bedrock/latest/userguide/create-automated-reasoning-policy.md +++ b//bedrock/latest/userguide/create-automated-reasoning-policy.md @@ -5 +5 @@ -Prepare your source documentWrite effective instructionsCreate a policy in the consoleCreate a policy using the APIReview the extracted policyIterative policy buildingKMS permissions for Automated Reasoning policies +Prepare your source documentWrite effective instructionsCreate a policy in the consoleCreate a policy using the APIReview the extracted policyReview the fidelity reportIterative policy buildingKMS permissions for Automated Reasoning policies @@ -233 +233 @@ After a build completes, review the extracted policy definition before you start -In the console, open your policy and go to the **Definitions** page. Via the API, use `GetAutomatedReasoningPolicyBuildWorkflowResultAssets` with `--asset-type POLICY_DEFINITION` to retrieve the extracted definition, and `--asset-type QUALITY_REPORT` to retrieve the quality report. +In the console, open your policy and go to the **Definitions** page. Via the API, use `GetAutomatedReasoningPolicyBuildWorkflowResultAssets` with `--asset-type POLICY_DEFINITION` to retrieve the extracted definition, and `--asset-type QUALITY_REPORT` to retrieve the quality report. You can see a full list of the assets produced during the workflow, such as the fidelity report, using the `--asset-type ASSET_MANIFEST` parameter. @@ -253,0 +254,44 @@ The quality report also identifies disjoint rule sets — groups of rules that d +## Review the fidelity report + +When you create a policy from a source document, a fidelity report is automatically generated alongside the extracted policy. The fidelity report measures how accurately the policy represents your source content and provides detailed grounding that links each rule and variable back to specific statements in the document. For more information about fidelity report concepts, see [Fidelity report](./automated-reasoning-checks-concepts.html#ar-concept-fidelity-report). + +### Review the fidelity report in the console + +In the console, open your policy and choose the **Source Document** tab (next to **Definitions**). The **Source Content** view displays each atomic statement extracted from your document as a numbered row in a table. Each row shows: + + * The statement number and extracted text. + + * The source **Document** the statement came from. + + * The number of **Rules** grounded by that statement. + + * The number of **Variables** grounded by that statement. + + + + +Use the **Rules** and **Variables** dropdown filters at the top of the table to focus on statements that ground a specific rule or variable. Use the search bar to find specific content within the extracted statements. + +If you edit the policy after the initial extraction — for example, by modifying rules or adding variables — choose the **Regenerate** button to update the fidelity report so it reflects your current policy definition. + +### Review the fidelity report using the API + +Use `GetAutomatedReasoningPolicyBuildWorkflowResultAssets` with `--asset-type FIDELITY_REPORT` to retrieve the fidelity report. To regenerate the report after making policy changes, use `StartAutomatedReasoningPolicyBuildWorkflow` with the build workflow type `GENERATE_FIDELITY_REPORT` and provide the source documents in the `generateFidelityReportContent` field. The workflow re-analyzes the documents against the current policy definition and produces a new fidelity report. You can also retrieve the original source documents from a previous build workflow using `--asset-type SOURCE_DOCUMENT` with the `--asset-id` parameter (obtain the asset ID from the asset manifest). + +### What to look for + +When reviewing the fidelity report from the APIs, pay attention to: + + * **Low coverage score.** A low coverage score indicates that significant portions of your source document were not captured in the policy. Look for statements with 0 rules and 0 variables in the source content view to identify which parts of the document were missed, and consider using iterative policy building to add the missing content. See Iterative policy building. + + * **Low accuracy score on individual rules.** Each rule has its own accuracy score and justification. Rules with low accuracy scores may not faithfully represent the source material. Use the **Rules** filter to isolate the grounding statements for a specific rule and compare them against the rule's formal logic to identify misinterpretations. + + * **Ungrounded rules or variables.** Rules or variables that lack grounding statements may have been inferred rather than directly extracted from the document. Verify that these are correct or remove them if they don't reflect your intent. + + + + +###### Tip + +The fidelity report is especially useful for collaboration with domain experts who authored the source document. Share the **Source Document** view with them so they can verify that the policy correctly captures their intent without needing to read the formal logic rules directly. +