AWS bedrock documentation change
Summary
Comprehensive restructuring of Automated Reasoning policy testing documentation. Changes include: new testing strategy explanation (scenarios vs QnA tests), revised workflow recommendations, updated console/API instructions, detailed test result interpretation guidelines, and troubleshooting tips.
Security assessment
The changes focus on improving testing methodology clarity and workflow efficiency. There's no evidence of addressing security vulnerabilities, incidents, or weaknesses. Updates emphasize validation accuracy and policy correctness without mentioning security fixes, threat mitigation, or vulnerability disclosures. The content remains focused on functional testing best practices.
Diff
diff --git a/bedrock/latest/userguide/test-automated-reasoning-policy.md b/bedrock/latest/userguide/test-automated-reasoning-policy.md index 40389f8dd..04f879a0f 100644 --- a//bedrock/latest/userguide/test-automated-reasoning-policy.md +++ b//bedrock/latest/userguide/test-automated-reasoning-policy.md @@ -5 +5 @@ -Create a test manually in the consoleGenerate tests automatically in the consoleRun tests in the consoleCreate a test manually using the APIGenerate tests automatically using the APIRun tests using the API +Testing strategy: scenarios vs. QnA testsGenerate test scenarios automatically in the consoleGenerate test scenarios automatically using the APICreate a QnA test manually in the consoleCreate a QnA test using the APIRun testsUnderstand test results @@ -9 +9 @@ Create a test manually in the consoleGenerate tests automatically in the console -You test a policy by sending natural language statements or QnAs for validation, then inspect Automated Reasoning checks feedback to ensure it is translating the input text using the right variables and that the rules it's validating against are correct. +Testing validates that your policy's rules are correct and that Automated Reasoning checks can accurately translate natural language into formal logic. You test a policy by sending natural language statements for validation, then inspecting the feedback to ensure the translation uses the right variables and that the rules produce the expected results. @@ -11 +11 @@ You test a policy by sending natural language statements or QnAs for validation, -You can create tests in two ways: manually defining question-and-answer (QnA) pairs or automatically generating test scenarios. QnAs are specific user questions with corresponding model responses that you write to test particular use cases, while test scenarios are logical situations automatically generated from your policy rules that may or may not be realistic in your application context. +There are two complementary testing approaches: generated scenarios and question-and-answer (QnA) tests. Each targets a different part of the validation pipeline. The recommended workflow is to start with scenarios to validate rule correctness, then add QnA tests to validate translation accuracy. @@ -13 +13 @@ You can create tests in two ways: manually defining question-and-answer (QnA) pa -###### Note +## Testing strategy: scenarios vs. QnA tests @@ -15 +15 @@ You can create tests in two ways: manually defining question-and-answer (QnA) pa -**Tutorial video:** For a step-by-step walkthrough of testing an Automated Reasoning policy, watch the following tutorial: +Automated Reasoning checks validate content in two steps: first, foundation models translate natural language into formal logic; then, mathematical techniques verify the logic against your policy rules. Each testing approach targets a different step in this pipeline. @@ -17 +17 @@ You can create tests in two ways: manually defining question-and-answer (QnA) pa -[Tutorial Demo 2 - Testing the Automated Reasoning policy](https://youtu.be/a_NLNMdm33M) +### Generated scenarios (test rule correctness) @@ -19 +19 @@ You can create tests in two ways: manually defining question-and-answer (QnA) pa -Tests should mimic the questions your application's users would ask and the responses they might get from a foundation model. Automated Reasoning assesses the prompt and response accuracy with respect to the rules in your Automated Reasoning policy. Automated Reasoning performs this validation in the following steps: +Generated scenarios test the _semantics encoded in your policy rules directly_. They remove the uncertainty of natural language translation from the equation, isolating whether the rules themselves are correct. @@ -21 +21 @@ Tests should mimic the questions your application's users would ask and the resp -###### Tip +Scenarios are generated from your policy rules and represent situations that are logically possible given those rules. They are sorted to surface the most likely-to-be-wrong scenarios first. For each scenario, you review the variable assignments and decide: @@ -23 +23 @@ Tests should mimic the questions your application's users would ask and the resp -**Best practice:** Create tests that cover both valid and invalid scenarios. For example, if your policy states "Employees need 1 year of service for parental leave," create tests for responses that correctly state this rule and tests for responses that incorrectly state a different requirement. + * **Thumbs up** — The scenario is realistic and should indeed be possible. Save it as a `SATISFIABLE` test. @@ -25 +25 @@ Tests should mimic the questions your application's users would ask and the resp - 1. Uses your test's question and answer along with your policy's variables and their descriptions to translate the natural language inputs into formal logic. + * **Thumbs down** — Something is off. The scenario shouldn't be possible given your domain knowledge. Provide natural language feedback explaining why, and Automated Reasoning checks will attempt to deduce the necessary rule changes. @@ -27 +26,0 @@ Tests should mimic the questions your application's users would ask and the resp - 2. Validates the translated logic against the policy by using sound mathematical techniques. @@ -30,0 +30 @@ Tests should mimic the questions your application's users would ask and the resp +**Example:** Your policy says full-time employees with 12+ months of tenure are eligible for parental leave. A generated scenario might show `isFullTime = true, tenureMonths = 3, eligibleForParentalLeave = true`. If this scenario shouldn't be possible (because 3 months is less than 12), you'd give it a thumbs down and explain that employees need at least 12 months of tenure. This indicates a missing or incorrect rule. @@ -32 +32 @@ Tests should mimic the questions your application's users would ask and the resp -###### Note +Use scenarios as your _first_ testing step. They help you catch rule issues before you invest time writing QnA tests. @@ -34 +34 @@ Tests should mimic the questions your application's users would ask and the resp -Automated Reasoning translates natural language into logic using AI techniques that cannot guarantee perfect accuracy. However good tests will help detect and fix possible inaccuracies in your Automated Reasoning policies. +### QnA tests (test translation accuracy) @@ -36 +36 @@ Automated Reasoning translates natural language into logic using AI techniques t -## Create a test manually in the console +QnA tests validate the _full pipeline end-to-end_ : natural language translation and rule validation together. They mimic real user interactions and catch translation issues that scenarios can't detect. @@ -38 +38 @@ Automated Reasoning translates natural language into logic using AI techniques t - 1. Go to the Automated Reasoning policy that you want to test (for example, **MyHrPolicy**). +Each QnA test consists of: @@ -40 +40 @@ Automated Reasoning translates natural language into logic using AI techniques t - 2. Choose **View tests** , then select **Add**. + * An **input** (optional) — The question a user might ask your application. @@ -42 +42 @@ Automated Reasoning translates natural language into logic using AI techniques t - 3. In the **Add tests** dialog, do the following: + * An **output** — The response your foundation model might generate. @@ -44 +44 @@ Automated Reasoning translates natural language into logic using AI techniques t - 1. Include an input (optional) and output. These represent the question a user might ask and the response your foundation model might provide - together forming a QnA pair that tests how your policy validates real user interactions. + * An **expected result** — The validation result you expect (for example, `VALID` or `INVALID`). @@ -46 +45,0 @@ Automated Reasoning translates natural language into logic using AI techniques t - 2. Choose the result you expect from the test (such as **Valid** or **Invalid**). @@ -48 +46,0 @@ Automated Reasoning translates natural language into logic using AI techniques t - 3. Select a **Confidence threshold** , which is the minimum confidence level for logic validation. @@ -50 +47,0 @@ Automated Reasoning translates natural language into logic using AI techniques t - 4. Select **Save** to create the test. @@ -51,0 +49,3 @@ Automated Reasoning translates natural language into logic using AI techniques t +**Example:** For the same parental leave policy, a QnA test might be: input = "I've been working here for 2 years full-time. Can I take parental leave?", output = "Yes, you are eligible for parental leave.", expected result = `VALID`. This tests whether Automated Reasoning checks correctly translates "2 years" to `tenureMonths = 24` and "full-time" to `isFullTime = true`. + +###### Tip @@ -52,0 +53 @@ Automated Reasoning translates natural language into logic using AI techniques t +Create tests that cover both valid and invalid scenarios. For example, if your policy states "Employees need 1 year of service for parental leave," create tests for responses that correctly state this rule _and_ tests for responses that incorrectly state a different requirement. @@ -53,0 +55 @@ Automated Reasoning translates natural language into logic using AI techniques t +### Recommended testing workflow @@ -55 +57 @@ Automated Reasoning translates natural language into logic using AI techniques t -###### Note + 1. **Generate and review scenarios.** Start here to validate that your rules are correct. Fix any rule issues before proceeding. @@ -57 +59 @@ Automated Reasoning translates natural language into logic using AI techniques t -When creating a test, the confidence threshold is optional. + 2. **Write QnA tests for key use cases.** Focus on the questions your users are most likely to ask and the responses your LLM is most likely to generate. Include edge cases and boundary conditions. @@ -59 +61 @@ When creating a test, the confidence threshold is optional. - * Automated Reasoning checks uses multiple large language models (LLMs) to translate natural language tests into findings. It returns only "confident" findings that are supported by a significant percentage of the LLM translations. The confidence threshold defines the minimum percentage of support needed for a translation to become a finding with a validity result. + 3. **Run all tests.** Check that both scenarios and QnA tests pass. @@ -61 +63 @@ When creating a test, the confidence threshold is optional. - * If there are one or more translated findings that are not supported by a sufficient percentage of LLM translations, Automated Reasoning checks will surface an additional "TRANSLATION_AMBIGUOUS" finding. This finding will contain information to highlight the differences between the disagreeing LLM translations. + 4. **Iterate.** If tests fail, determine whether the issue is in the rules (fix the policy) or in the translation (improve variable descriptions). For more information, see [Troubleshoot and refine your Automated Reasoning policy](./address-failed-automated-reasoning-tests.html). @@ -66 +68 @@ When creating a test, the confidence threshold is optional. -## Generate tests automatically in the console +## Generate test scenarios automatically in the console @@ -68 +70 @@ When creating a test, the confidence threshold is optional. - 1. Go to the Automated Reasoning policy that you want to test (for example, _MyHrPolicy_). + 1. Go to the Automated Reasoning policy that you want to test (for example, **MyHrPolicy**). @@ -72 +74 @@ When creating a test, the confidence threshold is optional. - 3. In the **Generate scenarios** dialog, review the generated scenario and related rules. Then do one of the following: + 3. In the **Generate scenarios** dialog, review the generated scenario and the related rules. Each scenario shows a set of variable assignments that are logically possible given your policy rules. Evaluate whether the scenario is realistic in your domain: @@ -74 +76 @@ When creating a test, the confidence threshold is optional. - * If you think the scenario could happen (also known as a _satisfiable_ scenario), select the thumbs up (yes). + * If the scenario could happen in your domain (it is _satisfiable_), select the thumbs up icon. This saves the scenario as a test that expects a `SATISFIABLE` result. @@ -76 +78 @@ When creating a test, the confidence threshold is optional. - * If not, select the thumbs down (no). You can also provide an annotation to explain why you think the scenario isn't possible. This is similar to leaving a comment in a document. + * If the scenario shouldn't be possible, select the thumbs down icon. Provide an annotation explaining why — for example, "Employees need at least 12 months of tenure for parental leave, but this scenario shows 3 months with eligibility." Automated Reasoning checks uses your feedback to deduce rule changes that would prevent this scenario. @@ -78 +80 @@ When creating a test, the confidence threshold is optional. - * If you want a different scenario to test, choose **Regenerate scenario**. + * If you want a different scenario, choose **Regenerate scenario**. @@ -82 +84 @@ When creating a test, the confidence threshold is optional. -If you want to inspect the formal logic version of the scenario, enable **Show SMT-LIB**. +To inspect the formal logic version of the scenario, enable **Show SMT-LIB**. This is useful for understanding exactly which rules and variable assignments are involved. @@ -84 +86 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 4. Select **Save and close** to save the test or **Save and add another** test. + 4. Select **Save and close** to save the test, or **Save and add another** to continue reviewing scenarios. @@ -86 +88 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 5. If you provided annotations to any of the tests, choose **Apply annotations**. Automated Reasoning will make changes to your policy based on your feedback. + 5. If you provided annotations (thumbs down feedback) to any scenarios, choose **Apply annotations**. Automated Reasoning checks will start a build workflow to apply the changes to your policy based on your feedback. @@ -88 +90 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 6. On the **Review policy changes** screen, review the changes to your policy's rules, variables, and variable types. Then select **Accept changes**. + 6. On the **Review policy changes** screen, review the proposed changes to your policy's rules, variables, and variable types. Then select **Accept changes**. @@ -93 +95 @@ If you want to inspect the formal logic version of the scenario, enable **Show S -## Run tests in the console +## Generate test scenarios automatically using the API @@ -95 +97 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 1. Go to the Automated Reasoning policy that you want to validate (for example, **MyHrPolicy**). +Use the `GetAutomatedReasoningPolicyNextScenario` API to fetch generated test scenarios based on your policy's rules. @@ -97 +99 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 2. Choose **View tests**. +`policyArn` (required) @@ -99 +100,0 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - 3. Do one of the following: @@ -101 +102,25 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - * To run all of your policy's tests, choose **Validate all tests**. +The ARN of the Automated Reasoning policy. + +`buildWorkflowId` (required) + + +The identifier of the build workflow for the generated scenarios. Retrieve the latest build workflow using the `ListAutomatedReasoningPolicyBuildWorkflows` API. + +**Example:** + + + aws bedrock get-automated-reasoning-policy-next-scenario \ + --policy-arn "arn:aws:bedrock:us-east-1:111122223333:automated-reasoning-policy/lnq5hhz70wgk" \ + --build-workflow-id d40fa7fc-351e-47d8-a338-53e4b3b1c690 + +The response includes a generated scenario with variable assignments and the related policy rules. Review the scenario and use the `CreateAutomatedReasoningPolicyTestCase` API to save it as a test, or use the annotation APIs to provide feedback if the scenario reveals a rule issue. + +## Create a QnA test manually in the console + + 1. Go to the Automated Reasoning policy that you want to test (for example, **MyHrPolicy**). + + 2. Choose **View tests** , then select **Add**. + + 3. In the **Add tests** dialog, do the following: + + 1. For **Input** (optional), enter the question a user might ask. For **Output** , enter the response your foundation model might provide. Together these form a QnA pair that tests how your policy validates real user interactions. @@ -103 +128 @@ If you want to inspect the formal logic version of the scenario, enable **Show S - * To run tests individually, select the **Action** button next to the test that you want to run and choose **Validate**. + 2. Choose the result you expect from the test (such as **Valid** or **Invalid**). @@ -104,0 +130 @@ If you want to inspect the formal logic version of the scenario, enable **Show S + 3. (Optional) Select a **Confidence threshold** , which is the minimum confidence level for logic validation. Automated Reasoning checks uses multiple LLMs to translate natural language into findings. It returns only findings supported by a significant percentage of the LLM translations. The confidence threshold defines the minimum percentage of support needed for a translation to become a finding with a validity result. Findings below the threshold are surfaced as `TRANSLATION_AMBIGUOUS`. @@ -105,0 +132 @@ If you want to inspect the formal logic version of the scenario, enable **Show S + 4. Select **Save** to create the test. @@ -108 +134,0 @@ If you want to inspect the formal logic version of the scenario, enable **Show S -## Create a test manually using the API @@ -110 +135,0 @@ If you want to inspect the formal logic version of the scenario, enable **Show S -You can use the `CreateAutomatedReasoningPolicyTestCase` API operation to create a test for your Automated Reasoning policy programmatically. @@ -112 +137 @@ You can use the `CreateAutomatedReasoningPolicyTestCase` API operation to create -### Request parameters +## Create a QnA test using the API @@ -114 +139 @@ You can use the `CreateAutomatedReasoningPolicyTestCase` API operation to create -The following parameters are required or optional when creating a test: +Use the `CreateAutomatedReasoningPolicyTestCase` API to create a test programmatically. @@ -119 +144 @@ The following parameters are required or optional when creating a test: -The Amazon Resource Name (ARN) of the Automated Reasoning policy for which to create the test. +The ARN of the Automated Reasoning policy. @@ -129 +154 @@ The input query or prompt that generated the content, such as the user question. -The output content that's validated by the Automated Reasoning policy. This represents the foundation model response that will be checked for accuracy. +The output content to validate — the foundation model response that will be checked for accuracy. @@ -134 +159 @@ The output content that's validated by the Automated Reasoning policy. This repr -The expected validation result for the test (for example, `VALID` or `INVALID`). The actual result of the test is selected by sorting findings in order of importance and selecting the worst result. The sorting order is: ambiguous, impossible, invalid, satisfiable, and valid. For example, a test that results in two valid and one impossible finding will have an aggregated result of impossible. +The expected validation result (for example, `VALID` or `INVALID`). The actual result is determined by sorting findings in order of severity and selecting the worst result. The severity order from worst to best is: `TRANSLATION_AMBIGUOUS`, `IMPOSSIBLE`, `INVALID`, `SATISFIABLE`, `VALID`. @@ -139 +164 @@ The expected validation result for the test (for example, `VALID` or `INVALID`). -The minimum confidence level for logic validation. Content that meets the threshold is considered a high-confidence finding that can be validated. +The minimum confidence level for logic validation. @@ -141,3 +166 @@ The minimum confidence level for logic validation. Content that meets the thresh -### Example - -The following example shows how to create a test for an Automated Reasoning policy using the AWS CLI: +**Example:** @@ -161 +184 @@ Example response: -## Generate tests automatically using the API +## Run tests @@ -163 +186 @@ Example response: -You can use the `GetAutomatedReasoningPolicyNextScenario` API operation to fetch the next generated test scenarios based on your policy's rules. +### Run tests in the console @@ -165 +188 @@ You can use the `GetAutomatedReasoningPolicyNextScenario` API operation to fetch -### Request parameters + 1. Go to the Automated Reasoning policy that you want to validate (for example, **MyHrPolicy**). @@ -167 +190 @@ You can use the `GetAutomatedReasoningPolicyNextScenario` API operation to fetch -The following parameters are required or optional when generating test scenarios: + 2. Choose **View tests**. @@ -169 +192 @@ The following parameters are required or optional when generating test scenarios