AWS bedrock documentation change
Summary
Added new section about preprocessing documents with LLMs before automated reasoning checks, including warnings about hallucinated rules and need for human review
Security assessment
The change improves policy implementation guidance but doesn't directly address security vulnerabilities or document security features. The LLM warning focuses on accuracy/precision rather than security risks, though it indirectly prevents potential policy misconfigurations.
Diff
diff --git a/bedrock/latest/userguide/automated-reasoning-policy-best-practices.md b/bedrock/latest/userguide/automated-reasoning-policy-best-practices.md index 85a81f913..bb00a170e 100644 --- a//bedrock/latest/userguide/automated-reasoning-policy-best-practices.md +++ b//bedrock/latest/userguide/automated-reasoning-policy-best-practices.md @@ -5 +5 @@ -Start simple and iterateUse implications (=>) to structure rulesWrite comprehensive variable descriptionsUse booleans for non-exclusive statesSpecify units and formats in variable descriptionsValidate ranges for numerical valuesUse intermediate variables for abstractionUse enums for categorizationKeep logic declarative, not proceduralNaming conventionsCommon anti-patterns +Start simple and iteratePre-process documents with an LLMUse implications (=>) to structure rulesWrite comprehensive variable descriptionsUse booleans for non-exclusive statesSpecify units and formats in variable descriptionsValidate ranges for numerical valuesUse intermediate variables for abstractionUse enums for categorizationKeep logic declarative, not proceduralNaming conventionsCommon anti-patterns @@ -29,0 +30,21 @@ This approach has two advantages: it makes issues easier to isolate (you know wh +## Pre-process documents with an LLM + +For documents that are lengthy, contain narrative prose, or mix rules with non-rule content (such as legal disclaimers or organizational background), run the document through an LLM before uploading it to Automated Reasoning checks. Ask the LLM to extract the content as explicit if-then rules. This preprocessing step significantly improves the quality of the extracted policy because Automated Reasoning checks works best with clear, declarative statements rather than unstructured text. + +When writing your preprocessing prompt, include the following instructions for the LLM: + + * Extract rules in if-then format with clear conditions and consequences. + + * Preserve all conditions, logical operators (AND, OR, NOT), quantifiers ("at least", "at most"), and exception clauses ("unless", "except when"). + + * Add sanity rules for common-sense constraints — such as "account balance cannot be negative" or "credit score must be between 300 and 850" — which translate into boundary rules in your policy (see Validate ranges for numerical values). + + + + +###### Important + +Always review the LLM's output against your original document before using it as source text. LLMs can hallucinate rules not present in the source, misinterpret conditions, or drop important exceptions. The preprocessing step is a starting point — not a substitute for human review. + +For detailed prompt templates and a step-by-step preprocessing workflow, see [(Optional) Use an LLM to rewrite documents as logical rules](./create-automated-reasoning-policy.html#preprocess-with-llm). +