AWS cli documentation change
Summary
Added new --evaluation-reference-inputs parameter for providing ground truth data during agent evaluation, including expected responses, assertions, and tool trajectories. Also added ignoredReferenceInputFields output field to track unused reference inputs.
Security assessment
This change adds functionality for evaluation and testing of AI agents by allowing ground truth data comparison. While evaluation can help identify security issues in AI responses, this is a feature enhancement for testing capabilities rather than addressing a specific security vulnerability. The change documents new CLI parameters for evaluation workflows.
Diff
diff --git a/cli/latest/reference/bedrock-agentcore/evaluate.md b/cli/latest/reference/bedrock-agentcore/evaluate.md index be7236748..d9b263122 100644 --- a//cli/latest/reference/bedrock-agentcore/evaluate.md +++ b//cli/latest/reference/bedrock-agentcore/evaluate.md @@ -15 +15 @@ - * [AWS CLI 2.34.19 Command Reference](../../index.html) » + * [AWS CLI 2.34.20 Command Reference](../../index.html) » @@ -71,0 +72 @@ See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/bedroc + [--evaluation-reference-inputs <value>] @@ -201,0 +203,147 @@ JSON Syntax: +`--evaluation-reference-inputs` (list) + +> Ground truth data to compare against agent responses during evaluation. Allows to provide expected responses, assertions, and expected tool trajectories at different evaluation levels. Session-level reference inputs apply to the entire conversation, while trace-level reference inputs target specific request-response interactions identified by trace ID. +> +> Constraints: +> +> * min: `1` +> * max: `1000` +> + +> +> (structure) +> +>> A reference input containing ground truth data for evaluation, scoped to a specific context level (session or trace) through its span context. +>> +>> context -> (tagged union structure) [required] +>> +>>> The contextual information associated with an evaluation, including span context details that identify the specific traces and sessions being evaluated within the agent’s execution flow. +>>> +>>> ### Note +>>> +>>> This is a Tagged Union structure. Only one of the following top level keys can be set: `spanContext`. +>>> +>>> spanContext -> (structure) +>>> +>>>> The span context information that uniquely identifies the trace and span being evaluated, including session ID, trace ID, and span ID for precise targeting within the agent’s execution flow. +>>>> +>>>> sessionId -> (string) [required] +>>>> +>>>>> The unique identifier of the session containing this span. Sessions represent complete conversation flows and are detected using configurable `SessionTimeoutMinutes` (default 15 minutes). +>>>> +>>>> traceId -> (string) +>>>> +>>>>> The unique identifier of the trace containing this span. Traces represent individual request-response interactions within a session and group related spans together. +>>>> +>>>> spanId -> (string) +>>>> +>>>>> The unique identifier of the specific span being referenced. Spans represent individual operations like tool calls, model invocations, or other discrete actions within the agent’s execution. +>> +>> expectedResponse -> (tagged union structure) +>> +>>> The expected response for trace-level evaluation. Built-in evaluators that support this field compare the agent’s actual response against this value for assessment. Custom evaluators can access it through the `{expected_response}` placeholder in their instructions. +>>> +>>> ### Note +>>> +>>> This is a Tagged Union structure. Only one of the following top level keys can be set: `text`. +>>> +>>> text -> (string) +>>> +>>>> The text content of the ground truth data. Used for expected response text and assertion statements. +>>>> +>>>> Constraints: +>>>> +>>>> * min: `1` +>>>> * max: `100000` +>>>> + +>> +>> assertions -> (list) +>> +>>> A list of assertion statements for session-level evaluation. Each assertion describes an expected behavior or outcome the agent should demonstrate during the session. +>>> +>>> Constraints: +>>> +>>> * min: `1` +>>> * max: `100` +>>> + +>>> +>>> (tagged union structure) +>>> +>>>> A content block for ground truth data in evaluation reference inputs. Supports text content for expected responses and assertions. +>>>> +>>>> ### Note +>>>> +>>>> This is a Tagged Union structure. Only one of the following top level keys can be set: `text`. +>>>> +>>>> text -> (string) +>>>> +>>>>> The text content of the ground truth data. Used for expected response text and assertion statements. +>>>>> +>>>>> Constraints: +>>>>> +>>>>> * min: `1` +>>>>> * max: `100000` +>>>>> + +>> +>> expectedTrajectory -> (structure) +>> +>>> The expected tool call sequence for session-level trajectory evaluation. Contains a list of tool names representing the tools the agent is expected to invoke. +>>> +>>> toolNames -> (list) +>>> +>>>> The list of tool names representing the expected tool call sequence. +>>>> +>>>> Constraints: +>>>> +>>>> * min: `0` +>>>> * max: `1000` +>>>> + +>>>> +>>>> (string) +>>>> +>>>>> Constraints: +>>>>> +>>>>> * min: `1` +>>>>> * max: `500` +>>>>> + + +Shorthand Syntax: + + + context={spanContext={sessionId=string,traceId=string,spanId=string}},expectedResponse={text=string},assertions=[{text=string},{text=string}],expectedTrajectory={toolNames=[string,string]} ... + + +JSON Syntax: + + + [ + { + "context": { + "spanContext": { + "sessionId": "string", + "traceId": "string", + "spanId": "string" + } + }, + "expectedResponse": { + "text": "string" + }, + "assertions": [ + { + "text": "string" + } + ... + ], + "expectedTrajectory": { + "toolNames": ["string", ...] + } + } + ... + ] + + @@ -432,0 +581,20 @@ evaluationResults -> (list) +>> +>> ignoredReferenceInputFields -> (list) +>> +>>> The list of reference input field names that were provided but not used by the evaluator. Helps identify which ground truth data was not consumed during evaluation. +>>> +>>> Constraints: +>>> +>>> * min: `0` +>>> * max: `100` +>>> + +>>> +>>> (string) +>>> +>>>> Constraints: +>>>> +>>>> * min: `1` +>>>> * max: `1000` +>>>> + @@ -444 +612 @@ evaluationResults -> (list) - * [AWS CLI 2.34.19 Command Reference](../../index.html) » + * [AWS CLI 2.34.20 Command Reference](../../index.html) »