AWS bedrock documentation change
Summary
Major restructuring and expansion of Automated Reasoning checks documentation. Added detailed sections on content evaluation logic, API-specific implementation requirements (ApplyGuardrail/Converse/InvokeModel), interpretation of findings, and clarification workflows. Reorganized content flow and moved sample application details to end.
Security assessment
The changes enhance documentation for Automated Reasoning checks - a security feature validating LLM responses against policy rules. No evidence addresses a specific vulnerability; instead, improvements clarify proper implementation to prevent misconfigurations that could lead to security gaps. Added warnings about silent validation failures improve security posture.
Diff
diff --git a/bedrock/latest/userguide/integrate-automated-reasoning-checks.md b/bedrock/latest/userguide/integrate-automated-reasoning-checks.md index 82c47d7f7..8baea2da1 100644 --- a//bedrock/latest/userguide/integrate-automated-reasoning-checks.md +++ b//bedrock/latest/userguide/integrate-automated-reasoning-checks.md @@ -7 +7 @@ -Integration overviewOpen-source rewriting chatbot sampleCall ApplyGuardrail with Automated Reasoning checksInterpret AR findings at runtimeHandle validation outcomes in your applicationRewrite invalid responses using AR feedbackAsk clarifying questionsBuild an audit trail +How Automated Reasoning checks evaluate contentIntegration overviewCall ApplyGuardrail with Automated Reasoning checksCall Converse with Automated Reasoning checksCall InvokeModel with Automated Reasoning checksInterpret Automated Reasoning findings at runtimeHandle validation outcomes in your applicationRewrite invalid responses using AR feedbackAsk clarifying questionsOpen-source rewriting chatbot sampleBuild an audit trail @@ -13 +13,38 @@ After you deploy your Automated Reasoning policy in a guardrail (see [Deploy you -Automated Reasoning checks operate in _detect mode_ only — they return findings and feedback rather than blocking content. Your application is responsible for deciding what to do with the findings: serve the response, rewrite it, ask for clarification, or fall back to a default behavior. +Automated Reasoning checks operate in _detect mode_ only – they return findings and feedback rather than blocking content. Your application is responsible for deciding what to do with the findings: serve the response, rewrite it, ask for clarification, or fall back to a default behavior. + +## How Automated Reasoning checks evaluate content + +Automated Reasoning checks translate the content that you submit into a logical _implication_ – an "if/then" relationship – and then check that implication against your policy rules. The translation produces two kinds of logical statements: + + * **Premises** – The "if" side: the antecedent conditions and scenario facts that set the context for reasoning (for example, a fact the user states about their situation, or a condition the response makes its answer depend on). Premises are optional. + + * **Claims** – The "then" side: the assertions to validate against your policy rules (typically the substantive statements in the model's response). + + + + +Automated Reasoning checks decide which statements are premises and which are claims when they translate the content – the split is not taken directly from the input fields. What you control through the API is whether each piece of content enters as _user-side_ input (a question or stated condition) or _agent-side_ input (a response to validate). The `query` qualifier marks content as user-side; `guardContent` (or untagged text) marks it as agent-side; `groundingSource` is ignored by Automated Reasoning checks. The translation then derives the premises and claims from the combined input. + +Whether you have to tag content, and how, depends on the API that you use. + +API | Tagging required? | How content enters Automated Reasoning checks +---|---|--- +`ApplyGuardrail` | No (optional) | All content that you pass is evaluated. Each content block's `qualifiers` set whether it enters as user-side (`query`) or agent-side (`guard_content`) input; a block with no qualifier defaults to agent-side. `ApplyGuardrail` does not append a model response on your behalf, so your content must include at least one agent-side (claim) block. +`Converse` (plain text only) | Yes | Plain `text` blocks are not tagged, so Automated Reasoning checks have no content to evaluate and are skipped (`automatedReasoningPolicyUnits: 0`). Use a `guardContent` block to opt in. +`Converse` (with `guardContent`) | Yes | Use `qualifiers` on `guardContent` blocks to mark user-side and agent-side content. The model's response is appended as an agent-side (claim) block automatically. +`InvokeModel` | Yes | Wrap input text in XML tags and set `tagSuffix` in the request configuration. The model's response is appended as an agent-side (claim) block automatically. + +### Key differences between APIs + + * On `Converse` and `InvokeModel`, the model's response is appended as an agent-side (claim) block automatically. As a result, an input that you tag only with `query` still runs Automated Reasoning checks – the response supplies the claim. + + * Omitting tags on `InvokeModel`, or sending only plain text on `Converse`, does not produce an error, but Automated Reasoning checks are not applied. The response indicates this with `automatedReasoningPolicyUnits: 0`. + + * On `InvokeModel`, text between guardrail XML tags that carries no qualifier defaults to agent-side (claim) content. + + * Automated Reasoning checks evaluate the response (the agent-side content); they do not run on a standalone `INPUT`-only evaluation. + + * With `ApplyGuardrail`, no model response is appended for you, so the content you submit must include at least one agent-side (claim) block. If it does not, the request returns a `ValidationException`. + + + @@ -20 +57 @@ At runtime, the integration follows this flow: - User question ──► LLM generates response ──► ApplyGuardrail validates response + User question ──► LLM generates response ──► Validate response @@ -45 +82 @@ Automated Reasoning findings are returned through any API that supports a Amazon - * `ApplyGuardrail` — Standalone validation API. Use this when you want to validate content independently of the LLM invocation. This is the recommended approach for Automated Reasoning checks because it gives you full control over what content is validated and when. + * `ApplyGuardrail` – Standalone validation API. Use this when you want to validate content independently of the LLM invocation. This is the recommended approach for Automated Reasoning checks because it gives you full control over what content is validated and when. @@ -47 +84 @@ Automated Reasoning findings are returned through any API that supports a Amazon - * `Converse` and `InvokeModel` — LLM invocation APIs with guardrail configuration. Automated Reasoning findings are returned in the `trace` field of the response. + * `Converse` and `InvokeModel` – LLM invocation APIs with guardrail configuration. Automated Reasoning findings are returned in the `trace` field of the response. @@ -49 +85,0 @@ Automated Reasoning findings are returned through any API that supports a Amazon - * `InvokeAgent` and `RetrieveAndGenerate` — Agent and knowledge base APIs with guardrail configuration. @@ -53,26 +88,0 @@ Automated Reasoning findings are returned through any API that supports a Amazon - -This page focuses on the `ApplyGuardrail` API because it provides the most flexibility for implementing the rewriting and clarification patterns described below. For information about using guardrails with the other APIs, see [Use a guardrail](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-use.html). - -## Open-source rewriting chatbot sample - -For a complete, production-style implementation of the patterns described on this page, see the [Automated Reasoning checks rewriting chatbot](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/responsible_ai/automated-reasoning-rewriting-chatbot) on GitHub. This sample application demonstrates: - - * An iterative rewriting loop where invalid responses are automatically corrected based on AR feedback. - - * Follow-up questions when the LLM needs additional context from the user to rewrite accurately. - - * A timeout mechanism that automatically resumes processing when users don't respond to clarification questions. - - * Policy context injection into LLM prompts so the LLM can reference the full policy rules during rewriting. - - * JSON audit logging of every validation iteration for compliance and debugging. - - - - -The sample uses a Python/Flask backend with a React frontend and communicates with Amazon Bedrock for LLM inference and Amazon Bedrock Guardrails for validation through the `ApplyGuardrail` API. - -###### Note - -The sample application includes the policy content directly in the LLM generation prompts to support any Automated Reasoning policy without requiring document uploads. In a production deployment, you would typically use RAG content or feed the LLM the original natural language document instead of the Automated Reasoning policy source code. - @@ -81 +91 @@ The sample application includes the policy content directly in the LLM generatio -Use the `ApplyGuardrail` API to validate content against your guardrail. The API accepts one or more content blocks and returns an assessment that includes Automated Reasoning findings. +`ApplyGuardrail` evaluates all the content that you pass. Tagging is optional: by default each content block is treated as agent-side (claim) content and validated against your policy rules, which is the simplest integration path. To give Automated Reasoning checks additional context, you can set `qualifiers` on a content block to mark it as user-side (`query`) input. Unlike `Converse` and `InvokeModel`, `ApplyGuardrail` does not append a model response for you, so the content you submit must include at least one claim block; otherwise the request returns a `ValidationException`. @@ -157 +167 @@ An array of content blocks to validate. Each block contains a `text` field with -The `ApplyGuardrail` response includes an `assessments` array. Each assessment contains an `automatedReasoningPolicy` object with a `findings` array. Each finding is a union type — exactly one of the following keys is present: +The `ApplyGuardrail` response includes an `assessments` array. Each assessment contains an `automatedReasoningPolicy` object with a `findings` array. Each finding is a union type – exactly one of the following keys is present: @@ -178 +188,179 @@ For a detailed description of each finding type and its fields, see [Findings an -## Interpret AR findings at runtime +## Call Converse with Automated Reasoning checks + +Automated Reasoning checks run on a `Converse` request only when the request includes at least one `guardContent` block. A request that sends only plain `text` does not run Automated Reasoning checks. + +### Plain text only (Automated Reasoning checks skipped) + +If your `Converse` request uses only plain `text` blocks (no `guardContent` blocks), the text is not tagged for guardrail evaluation, so Automated Reasoning checks have nothing to evaluate and are skipped: + + + { + "messages": [{"role": "user", "content": [{"text": "Apply a 20% discount to my order"}]}] + } + +This request returns `automatedReasoningPolicyUnits: 0`. Other guardrail policies (content, topic, word, and sensitive information) still evaluate the content; only Automated Reasoning checks are skipped. To run Automated Reasoning checks, use a `guardContent` block as shown in the next section. + +###### Important + +When Automated Reasoning checks are skipped, the request still succeeds with no error – so a misconfigured request can silently go unvalidated. Always confirm your configuration by checking that `automatedReasoningPolicyUnits` is greater than `0` in the response. A value of `0` means Automated Reasoning checks did not run (for example, because the content was not tagged), even though the request succeeded. + +### Using guardContent blocks + +To run Automated Reasoning checks on `Converse`, wrap the content you want evaluated in a `guardContent` block and set its qualifiers. This is how you mark which content is user-side and which is agent-side. + +###### Note + +The `Converse` API uses snake_case for qualifier strings (`guard_content`, `grounding_source`), while `InvokeModel` XML tags use camelCase (`guardContent`, `groundingSource`). These map to the same underlying roles. + +The following table shows how each qualifier marks content for Automated Reasoning checks. + +Qualifier string (snake_case) | XML tag equivalent (camelCase) | Automated Reasoning input role +---|---|--- +`"query"` | `query` | **User-side** – the user's question or stated conditions. Provides context that the translation can draw premises from. +`"guard_content"` | `guardContent` | **Agent-side** – content to validate against your policy. Supplies the claims the translation checks. +`"grounding_source"` | `groundingSource` | Ignored by Automated Reasoning checks (used by contextual grounding checks). + +The qualifier sets only whether content is user-side or agent-side; Automated Reasoning checks derive the actual premises and claims when they translate the combined input. A block with no qualifiers specified defaults to `guard_content` (agent-side). You can specify multiple qualifiers on one block. The precedence order is `guard_content > query > grounding_source`. + +###### Note + +On `Converse`, the model's response is appended as an agent-side (claim) block automatically, so a request whose blocks use only `query` still runs Automated Reasoning checks. (When you call `ApplyGuardrail` directly, no response is appended, so you must supply at least one agent-side block yourself or the request returns a `ValidationException`.) + +### Example: Call Converse with Automated Reasoning qualifiers using the AWS CLI + + + aws bedrock-runtime converse \ + --model-id "model-id" \ + --guardrail-config '{ + "guardrailIdentifier": "your-guardrail-id", + "guardrailVersion": "1", + "trace": "enabled" + }' \ + --messages '[ + { + "role": "user", + "content": [ + { + "guardContent": { + "text": { + "text": "Apply a 20% discount to my order and confirm it is done.", + "qualifiers": ["query"] + } + } + } + ] + } + ]' + +### When Automated Reasoning checks run + +The following table summarizes whether Automated Reasoning checks run for each `Converse` request shape. In every case where they run, the model's response is appended as an agent-side (claim) block. + +Request shape | Automated Reasoning checks run? +---|--- +Only plain `text` blocks, no `guardContent` | No – content is not tagged, so checks are skipped (`automatedReasoningPolicyUnits: 0`) +Has `guardContent` blocks with `guard_content` or no qualifiers | Yes +Has `guardContent` blocks with only `query` | Yes – the appended model response supplies the claim +Has `guardContent` blocks with only `grounding_source` | Yes – the appended model response supplies the claim. The `grounding_source` block itself is ignored (it contributes neither a premise nor a claim), but the request still runs because the response provides claim content. + +## Call InvokeModel with Automated Reasoning checks + +###### Warning + +You must tag your input with XML guardrail tags for Automated Reasoning checks to evaluate responses. Without tags, Automated Reasoning checks return `automatedReasoningPolicyUnits: 0` – no error is raised and no evaluation occurs. + +### How it works + +`InvokeModel` requires two things for Automated Reasoning checks: + + 1. A `tagSuffix` in the body's `amazon-bedrock-guardrailConfig` object. + + 2. Input text wrapped in XML tags that use that suffix. + + + + +The XML tag format is as follows: + + + <amazon-bedrock-guardrails-QUALIFIER_SUFFIX>text</amazon-bedrock-guardrails-QUALIFIER_SUFFIX> + +Where: + + * `QUALIFIER` is one of `query`, `guardContent`, or `groundingSource` (camelCase in XML). + + * `SUFFIX` matches the `tagSuffix` value in the request body. + + * `tagSuffix` must match the pattern `^[a-zA-Z0-9][a-zA-Z0-9-_]{0,18}[a-zA-Z0-9]$` (2–20 characters).