AWS bedrock documentation change
Summary
Complete restructuring of the troubleshooting guide for Automated Reasoning policies. Added detailed debugging workflow, translation issue resolution, rule correction methods, handling impossible results, annotation usage, quality reports, and Kiro CLI integration. Replaced brief sections with comprehensive diagnostic procedures and practical examples.
Security assessment
The changes are focused on improving debugging procedures and policy refinement workflows. There's no evidence of security vulnerabilities, incidents, or weaknesses being addressed. The content relates to general functionality improvements without security-specific context.
Diff
diff --git a/bedrock/latest/userguide/address-failed-automated-reasoning-tests.md b/bedrock/latest/userguide/address-failed-automated-reasoning-tests.md index 062b7e19d..ae1748f73 100644 --- a//bedrock/latest/userguide/address-failed-automated-reasoning-tests.md +++ b//bedrock/latest/userguide/address-failed-automated-reasoning-tests.md @@ -5 +5 @@ -Understanding annotationsWhen it's impossible to provide guidanceAutomated reasoning doesn't understand the source documentThe rules in your Automated Reasoning policy are wrongAutomated reasoning policy returns TRANSLATION_AMBIGUOUS +Debugging workflowFix translation issuesFix rule issuesFix impossible resultsUse annotations to repair your policyUse the quality reportUse Kiro CLI for policy refinement @@ -7 +7 @@ Understanding annotationsWhen it's impossible to provide guidanceAutomated reaso -# Address failed Automated Reasoning policy tests +# Troubleshoot and refine your Automated Reasoning policy @@ -9 +9,3 @@ Understanding annotationsWhen it's impossible to provide guidanceAutomated reaso -If a test of your Automated Reasoning policy fails, review the extracted logic and rules in the test findings. There are several ways to address the failure depending on the issue: +When an Automated Reasoning policy test fails — the actual result doesn't match the expected result — the issue is either in the translation (natural language was mapped to the wrong variables or values) or in the rules (the policy logic doesn't match your domain). This page provides a systematic approach to diagnosing and fixing both types of issues. + +Before you start troubleshooting, make sure you understand the two-step validation process (translate, then validate) described in [Translation: from natural language to formal logic](./automated-reasoning-checks-concepts.html#ar-concept-translation). This distinction is the key to efficient debugging. @@ -17 +19,132 @@ If a test of your Automated Reasoning policy fails, review the extracted logic a -## Understanding annotations +## Debugging workflow + +When a test fails, use the actual result to identify the type of issue and jump to the relevant section. + +Actual result | Likely cause | Where to look +---|---|--- +`TRANSLATION_AMBIGUOUS` | The translation models disagreed on how to interpret the input. Usually caused by overlapping variables, vague descriptions, or ambiguous input text. | Fix translation issues +`NO_TRANSLATIONS` | The input couldn't be mapped to any policy variables. Either the input is off-topic or the policy is missing variables for the concepts mentioned. | Fix translation issues +`TOO_COMPLEX` | The input or policy exceeds processing limits. Often caused by non-linear arithmetic or policies with too many interacting rules. | [Limitations and considerations](./guardrails-automated-reasoning-checks.html#automated-reasoning-limitations) +`IMPOSSIBLE` | The premises contradict each other, or the policy itself contains conflicting rules. | Fix impossible results +`VALID`, `INVALID`, or `SATISFIABLE` (but not what you expected) | Check the translation in the finding first. If the right variables are assigned with the right values, the issue is in your rules. If the translation is wrong, the issue is in your variable descriptions. | Translation wrong: Fix translation issues. Rules wrong: Fix rule issues. + +###### Tip + +Always check the translation first. In most cases, the mathematical validation (step 2) is correct — the issue is in how the natural language was translated to formal logic (step 1). Fixing variable descriptions is faster and less risky than changing rules. + +## Fix translation issues + +Translation issues occur when Automated Reasoning checks can't reliably map natural language to your policy's variables. The most visible symptom is a `TRANSLATION_AMBIGUOUS` result, but translation issues can also cause incorrect `VALID`, `INVALID`, or `SATISFIABLE` results when the wrong variables or values are assigned. + +### Diagnose TRANSLATION_AMBIGUOUS results + +A `TRANSLATION_AMBIGUOUS` finding includes two key fields that help you understand the disagreement: + + * `options` — The competing logical interpretations (up to 2). Each option contains its own translation with premises, claims, and confidence. Compare the options to see where the translation models disagreed. + + * `differenceScenarios` — Scenarios (up to 2) that illustrate how the different interpretations differ in meaning, with variable assignments highlighting the practical impact of the ambiguity. + + + + +Examine these fields to identify the specific source of ambiguity, then apply the appropriate fix from the following list. + +### Overlapping variable definitions + +When multiple variables could reasonably represent the same concept, the translation models disagree on which one to use. + +**Symptom:** The `options` in the `TRANSLATION_AMBIGUOUS` finding show the same concept assigned to different variables. For example, one option assigns "2 years of service" to `tenureMonths = 24` while the other assigns it to `monthsOfService = 24`. + +**Fix:** Merge the overlapping variables into a single variable with a comprehensive description. Update all rules that reference the deleted variable to use the remaining one. + +**Example:** + +Before (overlapping) | After (merged) +---|--- +`tenureMonths`: "How long the employee has worked in months." `monthsOfService`: "The employee's months of service." | `tenureMonths`: "The number of complete months the employee has been continuously employed. When users mention years of service, convert to months (for example, 2 years = 24 months). This variable captures all references to employment duration, length of service, time at the company, or seniority." (Delete `monthsOfService` and update rules.) + +### Incomplete variable descriptions + +Variable descriptions that lack detail about how users refer to concepts in everyday language make it difficult to map input to the correct variable. + +**Symptom:** The `options` show the correct variable but with different values, or the translation assigns a value that doesn't match what the user said. For example, "2 years" is translated to `tenureMonths = 2` instead of `tenureMonths = 24`. + +**Fix:** Update the variable description to include unit conversion rules, synonyms, and alternative phrasings. See [Write comprehensive variable descriptions](./automated-reasoning-policy-best-practices.html#bp-variable-descriptions) for detailed guidance. + +**Example:** + +Before (incomplete) | After (comprehensive) +---|--- +`isFullTime`: "Full-time status." | `isFullTime`: "Whether the employee works full-time (true) or part-time (false). Set to true when users mention being 'full-time', working 'full hours', or working 40+ hours per week. Set to false when users mention being 'part-time', working 'reduced hours', or working fewer than 40 hours per week." + +### Inconsistent value formatting + +Translation ambiguity can occur when the system is unsure how to format values such as numbers, dates, or percentages. + +**Symptom:** The `options` show the same variable but with different value formats. For example, one option translates "5%" to `interestRate = 5` while the other translates it to `interestRate = 0.05`. + +**Fix:** Update the variable description to specify the expected format and include conversion rules. See [Specify units and formats in variable descriptions](./automated-reasoning-policy-best-practices.html#bp-units-formats). + +### Ambiguous input text + +Sometimes the input itself is genuinely ambiguous — it contains vague pronouns, unclear references, or statements that can be interpreted multiple ways. + +**Symptom:** The `options` show fundamentally different interpretations of the same text. For example, "Can they take leave?" could refer to any employee type. + +**Fix:** If this is a test, rewrite the input to be more specific. At runtime, your application should ask the user for clarification when it receives a `TRANSLATION_AMBIGUOUS` result. For integration patterns, see [Integrate Automated Reasoning checks in your application](./integrate-automated-reasoning-checks.html). + +### Adjust the confidence threshold + +If you see `TRANSLATION_AMBIGUOUS` results for inputs that are borderline ambiguous, you can adjust the confidence threshold. Lowering the threshold allows translations with less model agreement to proceed to validation, reducing `TRANSLATION_AMBIGUOUS` results but increasing the risk of incorrect translations. + +###### Important + +Adjusting the threshold should be a last resort. In most cases, improving variable descriptions or removing overlapping variables is a better fix because it addresses the root cause. For more information on how thresholds work, see [Confidence thresholds](./automated-reasoning-checks-concepts.html#ar-concept-confidence-thresholds). + +## Fix rule issues + +Rule issues occur when the translation is correct but the policy logic doesn't match your domain. You've confirmed that the right variables are assigned with the right values, but the validation result is still wrong. + +### Getting VALID when you expected INVALID + +The policy doesn't have a rule that prohibits the claim. The response contradicts your domain knowledge, but the policy allows it. + +**Diagnosis:** Look at the `supportingRules` in the finding. These are the rules that prove the claim is valid. Check whether these rules are correct or whether a rule is missing. + +**Common causes and fixes:** + + * **Missing rule.** Your policy doesn't have a rule that covers this condition. Add a new rule that captures the constraint. For example, if the policy allows parental leave for all full-time employees but should require 12 months of tenure, add: `(=> (and isFullTime (<= tenureMonths 12)) (not eligibleForParentalLeave))` + + * **Rule is too permissive.** An existing rule allows more than it should. Edit the rule to add the missing condition. For example, change `(=> isFullTime eligibleForParentalLeave)` to `(=> (and isFullTime (> tenureMonths 12)) eligibleForParentalLeave)` + + * **Missing variable.** The policy doesn't have a variable to capture a relevant concept. Add the variable, write a clear description, and create rules that reference it. + + + + +### Getting INVALID when you expected VALID + +The policy has a rule that incorrectly prohibits the claim. + +**Diagnosis:** Look at the `contradictingRules` in the finding. These are the rules that disprove the claim. Check whether these rules are correct. + +**Common causes and fixes:** + + * **Rule is too restrictive.** An existing rule blocks a valid scenario. Edit the rule to relax the condition or add an exception. For example, if the rule requires 24 months of tenure but the policy should require only 12, update the threshold. + + * **Rule was misextracted.** Automated Reasoning checks misinterpreted your source document. Edit the rule to match the intended logic, or delete it and add a correct rule manually. + + + + +### Getting SATISFIABLE when you expected VALID + +The response is correct under some conditions but not all. The policy has additional rules that the response doesn't address. + +**Diagnosis:** Compare the `claimsTrueScenario` and `claimsFalseScenario` in the finding. The difference between them shows the conditions that the response doesn't mention. + +**Common causes and fixes:** + + * **Response is incomplete.** The test output doesn't mention all the conditions required by the policy. Update the test output to include the missing conditions, or change the expected result to `SATISFIABLE` if incomplete responses are acceptable for your use case. + + * **Policy has unnecessary rules.** The policy requires conditions that aren't relevant to this scenario. Review whether the additional rules should apply and remove them if they don't. @@ -19 +151,0 @@ If a test of your Automated Reasoning policy fails, review the extracted logic a -Annotations are corrections you apply to repair your policy when tests fail. If a test doesn't return your expected result, you can modify the test conditions, rerun it, and apply the successful modification as an annotation to update your policy. @@ -21 +152,0 @@ Annotations are corrections you apply to repair your policy when tests fail. If -Use annotations to: @@ -23 +153,0 @@ Use annotations to: - * Fix incorrect rules @@ -25 +155 @@ Use annotations to: - * Add missing variables +## Fix impossible results @@ -27 +157 @@ Use annotations to: - * Improve variable descriptions +An `IMPOSSIBLE` result means Automated Reasoning checks can't evaluate the claims because the premises are contradictory or the policy itself contains conflicting rules. There are two distinct causes. @@ -29 +159 @@ Use annotations to: - * Resolve translation ambiguities +### Contradictions in the input @@ -30,0 +161 @@ Use annotations to: +The test input contains statements that contradict each other. For example, "I'm a full-time employee and also part-time" sets `isFullTime = true` and `isFullTime = false` simultaneously, which is logically impossible. @@ -31,0 +163 @@ Use annotations to: +**Diagnosis:** Inspect the `translation` premises in the finding. Look for variables that are assigned contradictory values. @@ -32,0 +165 @@ Use annotations to: +**Fix:** If this is a test, rewrite the input to remove the contradiction. At runtime, your application should handle `IMPOSSIBLE` results by asking the user to clarify their input. @@ -34 +167 @@ Use annotations to: -### Example: Policy repair annotation +### Conflicts in the policy @@ -36 +169 @@ Use annotations to: -**Problem:** Policy approved leave for all full time employees, but source document requires 1+ years. +The policy contains rules that contradict each other, making it impossible for Automated Reasoning checks to reach a conclusion for inputs that involve the conflicting rules. @@ -38 +171 @@ Use annotations to: -**Original rule:** +**Diagnosis:** If the input is valid (no contradictory premises), the issue is in the policy. Check the `contradictingRules` field in the finding to identify which rules conflict. Also check the quality report (see Use the quality report) — it flags conflicting rules automatically. @@ -39,0 +173 @@ Use annotations to: +**Common causes and fixes:** @@ -41 +175 @@ Use annotations to: - if isFullTime is true, then eligibleForParentalLeave is true + * **Contradictory rules.** Two rules reach opposite conclusions for the same conditions. For example, one rule says full-time employees are eligible for leave, while another says employees in their first year are not eligible, without specifying what happens to full-time employees in their first year. Merge the rules into a single rule with explicit conditions: `(=> (and isFullTime (> tenureMonths 12)) eligibleForLeave)` @@ -43 +177 @@ Use annotations to: -**Annotation applied:** + * **Bare assertions.** A bare assertion like `(= eligibleForLeave true)` makes it impossible for any input to claim the user is _not_ eligible. Rewrite bare assertions as implications. See [Use implications (=>) to structure rules](./automated-reasoning-policy-best-practices.html#bp-use-implications). @@ -45 +179 @@ Use annotations to: - 1. Added `tenureMonths` variable (INT type) + * **Circular dependencies.** Rules that depend on each other in a way that creates logical loops. Simplify the rules to break the cycle, or use intermediate variables to make the logic explicit. @@ -47 +180,0 @@ Use annotations to: - 2. Updated rule to: `if isFullTime is true and tenureMonths is greater than 12, then eligibleForParentalLeave is true` @@ -49 +181,0 @@ Use annotations to: - 3. Test now correctly returns INVALID for employees with less than 12 months of tenure @@ -51,0 +184 @@ Use annotations to: +## Use annotations to repair your policy @@ -52,0 +186 @@ Use annotations to: +Annotations are targeted corrections you apply to your policy when tests fail. Instead of manually editing rules and variables, you can use annotations to describe the change you want and let Automated Reasoning checks apply it. Annotations are available through both the console and the API. @@ -54 +188 @@ Use annotations to: - * Update the failed test's conditions and rerun it. If you the test returns the validation result you expect, you can apply this annotation to update your policy.