AWS bedrock-agentcore documentation change
Summary
Complete restructuring of the Dataset Evaluations documentation. Removed detailed implementation examples and replaced with overview of two evaluation runners (On-demand and Batch), comparison table, prerequisites, and topic links to new dedicated pages.
Security assessment
The changes are purely organizational and functional improvements to documentation structure. No security vulnerabilities, configurations, or incidents are mentioned. The content focuses on evaluation workflow optimization without security implications.
Diff
diff --git a/bedrock-agentcore/latest/devguide/dataset-evaluations.md b/bedrock-agentcore/latest/devguide/dataset-evaluations.md index f1ef6dab5..7e101867b 100644 --- a//bedrock-agentcore/latest/devguide/dataset-evaluations.md +++ b//bedrock-agentcore/latest/devguide/dataset-evaluations.md @@ -7 +7 @@ -How it worksPrerequisitesDataset schemaSingle-turn exampleMulti-turn exampleInline dataset constructionComponents referenceResult structure +# Dataset evaluation @@ -9 +9 @@ How it worksPrerequisitesDataset schemaSingle-turn exampleMulti-turn exampleInli -# Dataset evaluations +Dataset evaluations let you run your agent against a set of scenarios and automatically evaluate the results. Instead of manually invoking your agent, collecting spans, and calling the Evaluate API, a dataset runner orchestrates the entire lifecycle in a single call: invoke the agent, wait for telemetry ingestion, and evaluate. @@ -11 +11 @@ How it worksPrerequisitesDataset schemaSingle-turn exampleMulti-turn exampleInli -Dataset evaluations let you run your agent against a predefined set of scenarios and automatically evaluate the results. Instead of manually invoking your agent and collecting spans, the `OnDemandEvaluationDatasetRunner` from the AgentCore SDK orchestrates the entire lifecycle — invoke the agent, wait for telemetry ingestion, collect spans, and call the Evaluate API — in a single `run()` call. +This is useful for regression testing, benchmark datasets, CI/CD pipelines, baseline measurement, and pre/post comparison after configuration changes. @@ -13 +13 @@ Dataset evaluations let you run your agent against a predefined set of scenarios -This is useful for regression testing, benchmark datasets, and CI/CD pipelines where you want to evaluate agent quality across many scenarios automatically. +The AgentCore SDK provides two dataset runners that share the same [dataset schema](./dataset-evaluations-schema.html) and ground truth format but differ in where evaluation happens: @@ -15,21 +15 @@ This is useful for regression testing, benchmark datasets, and CI/CD pipelines w -###### Note - -Dataset evaluations support all AgentCore evaluators — all built-in evaluators across session, trace, and tool-call levels, as well as custom evaluators. The runner automatically handles level-aware request construction, batching, and ground truth mapping for whichever evaluators you configure. - -###### Topics - - * How it works - - * Prerequisites - - * Dataset schema - - * Single-turn example - - * Multi-turn example - - * Inline dataset construction - - * Components reference - - * Result structure + * **On-demand dataset runner** (`OnDemandEvaluationDatasetRunner`) — Collects spans and calls the Evaluate API client-side. Best for dev-time iteration and small datasets. @@ -36,0 +17 @@ Dataset evaluations support all AgentCore evaluators — all built-in evaluators + * **Batch dataset runner** (`BatchEvaluationRunner`) — Delegates span collection and evaluation to the service via the batch evaluation API. Best for large datasets and production baselines. @@ -40,9 +20,0 @@ Dataset evaluations support all AgentCore evaluators — all built-in evaluators -## How it works - -The runner processes scenarios in three phases: - - 1. **Invoke** — All scenarios run concurrently using a thread pool. Each scenario gets a unique session ID, and turns within a scenario execute sequentially to maintain conversation context. - - 2. **Wait** — A configurable delay (default: 180 seconds) allows CloudWatch to ingest the telemetry data. This delay is paid once, not per-scenario. - - 3. **Evaluate** — Spans are collected from CloudWatch and evaluation requests are built for each evaluator. Ground truth fields from the dataset ( `expected_response` , `assertions` , `expected_trajectory` ) are automatically mapped to the correct API reference inputs. @@ -49,0 +22 @@ The runner processes scenarios in three phases: +**Choosing a runner** @@ -50,0 +24,7 @@ The runner processes scenarios in three phases: +Aspect | On-demand runner | Batch runner +---|---|--- +Span collection | SDK-side via `AgentSpanCollector` | Server-side; service reads from CloudWatch directly +Evaluate API calls | SDK calls `evaluate()` per evaluator per scenario | SDK calls `startBatchEvaluation()` once +Execution model | Synchronous three-phase pipeline (invoke, wait, evaluate) | Asynchronous four-phase pipeline (invoke, wait, submit, poll) +Results | Structured `EvaluationResult` with per-scenario, per-evaluator detail | Aggregate `BatchEvaluationSummary` with per-evaluator averages, plus per-session detail in CloudWatch +Best for | Dev-time iteration, CI/CD, small datasets, when you need per-scenario detail immediately | Baseline measurement, large datasets, pre/post comparison, when aggregate scores are sufficient @@ -51,0 +32 @@ The runner processes scenarios in three phases: +**Prerequisites** @@ -53 +34 @@ The runner processes scenarios in three phases: -## Prerequisites +Both runners require: @@ -63 +44 @@ The runner processes scenarios in three phases: - * Transaction Search enabled in CloudWatch — see [Enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) + * Transaction Search enabled in CloudWatch; see [Enable Transaction Search](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Transaction-Search-getting-started.html) @@ -72,308 +53 @@ The runner processes scenarios in three phases: -The following constants are used throughout the examples. Replace them with your own values: - - - REGION = "<region-code>" - AGENT_ARN = "arn:aws:bedrock-agentcore:<region-code>:<account-id>:runtime/<agent-id>" - LOG_GROUP = "/aws/bedrock-agentcore/runtimes/<agent-id>-DEFAULT" - -## Dataset schema - -A dataset contains one or more scenarios. Each scenario represents a conversation (session) with the agent. Scenarios can be single-turn or multi-turn. - - - { - "scenarios": [ - { - "scenario_id": "math-question", - "turns": [ - { - "input": "What is 15 + 27?", - "expected_response": "15 + 27 = 42" - } - ], - "expected_trajectory": ["calculator"], - "assertions": ["Agent used the calculator tool to compute the result"] - } - ] - } - -Field | Required | Scope | Description ----|---|---|--- -`scenario_id` | Yes | — | Unique identifier for the scenario. -`turns` | Yes | — | List of turns in the conversation. Each turn has `input` (required) and `expected_response` (optional). -`expected_trajectory` | No | Session | Expected sequence of tool names. Used by trajectory evaluators. -`assertions` | No | Session | Natural language assertions about expected behavior. Used by `Builtin.GoalSuccessRate`. - -Field | Required | Description ----|---|--- -`input` | Yes | The prompt sent to the agent for this turn. Can be a string or a dict. -`expected_response` | No | The expected agent response for this turn. Mapped positionally to the trace produced by this turn. - -The runner automatically maps dataset fields to the Evaluate API’s `evaluationReferenceInputs` : - - * `expected_response` on each turn maps positionally to traces — turn 0 → trace 0, turn 1 → trace 1, and so on. - - * `assertions` and `expected_trajectory` are scoped to the session level. - - * If no ground truth fields are present, `evaluationReferenceInputs` is omitted from the API request. - - - - -## Single-turn example - -A single-turn dataset has one turn per scenario. This is the simplest form — each scenario sends one prompt and checks the response. - -Save the following as `dataset.json` : - - - { - "scenarios": [ - { - "scenario_id": "math-question", - "turns": [ - { - "input": "What is 15 + 27?", - "expected_response": "15 + 27 = 42" - } - ], - "expected_trajectory": ["calculator"], - "assertions": ["Agent used the calculator tool to compute the result"] - }, - { - "scenario_id": "weather-check", - "turns": [ - { - "input": "What's the weather?", - "expected_response": "The weather is sunny" - } - ], - "expected_trajectory": ["weather"], - "assertions": ["Agent used the weather tool"] - } - ] - } - -Run the evaluation: - - - import json - import boto3 - from bedrock_agentcore.evaluation import ( - OnDemandEvaluationDatasetRunner, - EvaluationRunConfig, - EvaluatorConfig, - FileDatasetProvider, - CloudWatchAgentSpanCollector, - AgentInvokerInput, - AgentInvokerOutput, - ) - - # Load dataset - dataset = FileDatasetProvider("dataset.json").get_dataset() - - # Define the agent invoker - agentcore_client = boto3.client("bedrock-agentcore", region_name=REGION) - - def agent_invoker(invoker_input: AgentInvokerInput) -> AgentInvokerOutput: - payload = invoker_input.payload - if isinstance(payload, str): - payload = json.dumps({"prompt": payload}).encode() - elif isinstance(payload, dict): - payload = json.dumps(payload).encode() - - response = agentcore_client.invoke_agent_runtime( - agentRuntimeArn=AGENT_ARN, - runtimeSessionId=invoker_input.session_id, - payload=payload, - ) - response_body = response["response"].read() - return AgentInvokerOutput(agent_output=json.loads(response_body)) - - # Create span collector - span_collector = CloudWatchAgentSpanCollector( - log_group_name=LOG_GROUP, - region=REGION, - ) - - # Configure evaluators - config = EvaluationRunConfig( - evaluator_config=EvaluatorConfig(