AWS bedrock-agentcore documentation change
Summary
Added new 'Batch evaluation' section explaining asynchronous session evaluation with CloudWatch Logs integration
Security assessment
The change introduces a new evaluation feature without any security context or evidence of addressing security vulnerabilities.
Diff
diff --git a/bedrock-agentcore/latest/devguide/evaluations-types.md b/bedrock-agentcore/latest/devguide/evaluations-types.md index a3cb85688..8f55ff814 100644 --- a//bedrock-agentcore/latest/devguide/evaluations-types.md +++ b//bedrock-agentcore/latest/devguide/evaluations-types.md @@ -7 +7 @@ -Online evaluationOn-demand evaluation +Online evaluationOn-demand evaluationBatch evaluation @@ -11 +11 @@ Online evaluationOn-demand evaluation -AgentCore Evaluations provides two evaluation types, which differ in when and how the evaluation is performed: +AgentCore Evaluations provides three evaluation types, which differ in when and how the evaluation is performed: @@ -18,0 +19,2 @@ AgentCore Evaluations provides two evaluation types, which differ in when and ho + * Batch evaluation + @@ -35,0 +38,8 @@ This evaluation type complements online evaluation by offering precise control o +## Batch evaluation + +Batch evaluation runs evaluators against multiple agent sessions in a single asynchronous job. Unlike on-demand evaluation where you collect spans and call the Evaluate API per session, batch evaluation handles session discovery, span collection, and scoring entirely on the service side. You submit a job specifying the CloudWatch Logs location of your agent sessions and which evaluators to run. The service processes all matching sessions and returns aggregate results with per-evaluator average scores. + +Batch evaluation supports ground truth through session metadata, enabling reference-based scoring with expected responses, assertions, and expected tool trajectories. Results include both aggregate summaries (per-evaluator averages and session counts) and per-session detail written to CloudWatch Logs. + +This evaluation type is designed for baseline measurement before making changes, pre/post comparison after applying prompt or model updates, regression testing across curated session sets, and periodic quality audits across production traffic from a specific time window. +