AWS bedrock-agentcore documentation change

Service: bedrock-agentcore · 2026-04-25 · Documentation low

File: bedrock-agentcore/latest/devguide/code-based-evaluators.md

Summary

Added documentation for running online evaluations with code-based evaluators, including new Lambda payload field, CLI/SDK examples, and ground truth reference inputs functionality

Security assessment

This change adds new features for online evaluation configurations and ground truth functionality. While it mentions IAM permissions and execution roles, these are standard AWS security practices for Lambda functions and evaluation roles, not addressing any specific security vulnerability or incident. The change is primarily feature enhancement and documentation expansion.

Diff

diff --git a/bedrock-agentcore/latest/devguide/code-based-evaluators.md b/bedrock-agentcore/latest/devguide/code-based-evaluators.md
index e1ba1424e..3ceab5842 100644
--- a//bedrock-agentcore/latest/devguide/code-based-evaluators.md
+++ b//bedrock-agentcore/latest/devguide/code-based-evaluators.md
@@ -7 +7 @@
-PrerequisitesIAM permissionsLambda function contractCreate a code-based evaluatorRun on-demand evaluation with a code-based evaluator
+PrerequisitesIAM permissionsLambda function contractCreate a code-based evaluatorRun on-demand evaluation with a code-based evaluatorRun online evaluation with code-based evaluator
@@ -59,0 +60 @@ Your Lambda function receives a JSON payload with the following structure:
+        "evaluationReferenceInputs": [],
@@ -73,0 +75 @@ Field | Type | Description
+`evaluationReferenceInputs` |  List |  Reference inputs provided to the evaluator, filtered based on the evaluation level. See Using ground truth in code-based evaluator.  
@@ -136,0 +139,12 @@ The following code samples demonstrate how to create code-based evaluators using
+AgentCore CLI
+    
+
+  1.     agentcore eval evaluator create \
+      --name "MyCodeEvaluator" \
+      --level TRACE \
+      --lambda-arn "arn:aws:lambda:us-east-1:123456789012:function:my-eval-function" \
+      --lambda-timeout 120
+
+
+
+
@@ -223,0 +238,11 @@ Once created, use the custom code-based evaluator with the `Evaluate` API the sa
+AgentCore CLI
+    
+
+  1.     agentcore run eval \
+      --runtime "your_runtime_name" \
+      --session-id "your_session_id" \
+      --evaluator "code-based-evaluator-id"
+
+
+
+
@@ -293,0 +319,103 @@ You can target specific traces or spans, just like with LLM-based evaluators:
+### Using ground truth in code-based evaluator
+
+When ground truth reference inputs are configured, your Lambda function receives them in the `evaluationReferenceInputs` field. The reference inputs included depend on the evaluation level:
+
+Evaluation level | Lambda receives  
+---|---  
+`SESSION` |  All reference inputs.  
+`TRACE` |  Session-level reference inputs plus reference inputs matching the target traceId.  
+`TOOL_CALL` |  Session-level reference inputs plus reference inputs matching the target spanId.  
+  
+###### Note
+
+For more information about using ground truth evaluations, see [Ground truth evaluations](./ground-truth-evaluations.html).
+
+## Run online evaluation with code-based evaluator
+
+You can use a custom code-based evaluator in an online evaluation configuration to continuously monitor your agent’s live traffic. Pass the evaluator ID in the `evaluators` list when calling `CreateOnlineEvaluationConfig`.
+
+###### Example
+
+AgentCore CLI
+    
+
+  1.     agentcore add online-eval \
+      --name "your_config_name" \
+      --runtime "your_runtime_name" \
+      --evaluator "code-based-evaluator-id" \
+      --sampling-rate 1.0 \
+      --enable-on-create
+
+This command adds the online evaluation configuration to your local `agentcore.json` . Run `agentcore deploy` to create it in your AWS account.
+
+###### Note
+
+Run this from inside an AgentCore project directory (created with `agentcore create` ).
+
+
+
+
+AgentCore SDK
+    
+
+  1.     from bedrock_agentcore_starter_toolkit import Evaluation
+    
+    eval_client = Evaluation()
+    
+    config = eval_client.create_online_config(
+        config_name="my_online_eval_config",
+        agent_id="agent-id",
+        sampling_rate=1.0,
+        evaluator_list=["code-based-evaluator-id"],
+        enable_on_create=True
+    )
+    
+    print(f"Config ID: {config['onlineEvaluationConfigId']}")
+
+
+
+
+AWS SDK
+    
+
+  1.     import boto3
+    
+    client = boto3.client('bedrock-agentcore-control')
+    
+    response = client.create_online_evaluation_config(
+        onlineEvaluationConfigName="my_online_eval_config",
+        rule={"samplingConfig": {"samplingPercentage": 100.0}},
+        dataSourceConfig={
+            "cloudWatchLogs": {
+                "logGroupNames": ["/aws/agentcore/my-agent-traces"],
+                "serviceNames": ["my-agent.DEFAULT"]
+            }
+        },
+        evaluators=[{"evaluatorId": "code-based-evaluator-id"}],
+        evaluationExecutionRoleArn="arn:aws:iam::account-id:role/AgentCoreEvaluationRole",
+        enableOnCreate=True
+    )
+    
+    print(f"Config ID: {response['onlineEvaluationConfigId']}")
+
+
+
+
+AWS CLI
+    
+
+  1.     aws bedrock-agentcore-control create-online-evaluation-config \
+        --online-evaluation-config-name "my_online_eval_config" \
+        --rule '{"samplingConfig": {"samplingPercentage": 100.0}}' \
+        --data-source-config '{"cloudWatchLogs": {"logGroupNames": ["/aws/agentcore/my-agent-traces"], "serviceNames": ["my-agent.DEFAULT"]}}' \
+        --evaluators '[{"evaluatorId": "code-based-evaluator-id"}]' \
+        --evaluation-execution-role-arn "arn:aws:iam::account-id:role/AgentCoreEvaluationRole" \
+        --enable-on-create
+
+
+
+
+###### Note
+
+When an online evaluation configuration referencing a code-based evaluator is enabled, the evaluator is automatically locked and cannot be modified or deleted until the configuration is disabled or deleted. To make changes to the evaluator, disable the online evaluation configuration first, or clone the evaluator and create a new configuration.
+