AWS bedrock-agentcore documentation change
Summary
Restructured A/B testing workflow: Added explicit gateway/target creation step, changed CLI commands from 'add' to 'run' for tests, introduced job IDs for test management, updated verification and traffic generation procedures, and renumbered all steps.
Security assessment
The changes are procedural improvements to A/B testing implementation, focusing on workflow clarity and updated CLI syntax. No security vulnerabilities, patches, or security-specific features are mentioned. Modifications relate to test setup, execution patterns, and result management without security context.
Diff
diff --git a/bedrock-agentcore/latest/devguide/ab-testing-target-based.md b/bedrock-agentcore/latest/devguide/ab-testing-target-based.md index 2dae24bcc..5f4d52313 100644 --- a//bedrock-agentcore/latest/devguide/ab-testing-target-based.md +++ b//bedrock-agentcore/latest/devguide/ab-testing-target-based.md @@ -7 +7 @@ -Step 1: Create the projectStep 2: Add the runtimeStep 3: Deploy control and treatment versionsStep 4: Create online evaluation configurationsStep 5: Create the A/B testStep 6: Send traffic through the AgentCore GatewayStep 7: Get resultsStep 8: Confirm results and stop the A/B testStep 9: Deploy the winnerUnderstanding resultsTroubleshooting +Step 1: Create the projectStep 2: Add the runtimeStep 3: Deploy control and treatment versionsStep 4: Create online evaluation configurationsStep 5: Create the gateway and targetsStep 6: Create the A/B testStep 7: Send traffic through the AgentCore GatewayStep 8: Get resultsStep 9: Confirm results and stop the A/B testStep 10: Deploy the winnerUnderstanding resultsTroubleshooting @@ -25,0 +26,4 @@ This walkthrough deploys two versions of the customer support agent — one usin +###### Note + +This walkthrough is for agents hosted on an AgentCore Runtime. If your agent runs **outside** an AgentCore Runtime (a third-party or self-hosted agent — for example on AWS Lambda), see [Run an A/B test for agents hosted outside of AgentCore](./ab-testing-3p-agents.html) instead. + @@ -168 +172 @@ You now have: -Verify both endpoints are working: +Verify the runtime is working: @@ -171,2 +175 @@ Verify both endpoints are working: - agentcore invoke --runtime csAgent --target control --prompt "What is the status of order ORD-1003?" - agentcore invoke --runtime csAgent --target treatment --prompt "What is the status of order ORD-1003?" + agentcore invoke --runtime csAgent --prompt "What is the status of order ORD-1003?" @@ -210 +213,3 @@ For more details on evaluator options and configuration, see [Create online eval -## Step 5: Create the A/B test +## Step 5: Create the gateway and targets + +A target-based A/B test routes traffic through an AgentCore Gateway, so the gateway and its two targets must already be deployed before you start the test. Add a gateway and register each runtime endpoint as an `http-runtime` target, then deploy: @@ -212 +217,22 @@ For more details on evaluator options and configuration, see [Create online eval -Create an A/B test that routes traffic between the two endpoints. Each variant references an endpoint as an AgentCore Gateway target and has its own online evaluation config. The CLI registers both endpoints as AgentCore Gateway targets and creates the AgentCore Gateway automatically. + + agentcore add gateway --name csGateway + + agentcore add gateway-target \ + --name customer-support-control \ + --gateway csGateway \ + --type http-runtime \ + --runtime csAgent \ + --runtime-endpoint control + + agentcore add gateway-target \ + --name customer-support-treatment \ + --gateway csGateway \ + --type http-runtime \ + --runtime csAgent \ + --runtime-endpoint treatment + + agentcore deploy + +## Step 6: Create the A/B test + +Start the A/B test with `agentcore run ab-test`. Each variant references one of the gateway targets you created and has its own online evaluation config. The command initiates the test directly on the service against the already-deployed gateway. @@ -220 +246 @@ AgentCore CLI - agentcore add ab-test \ + agentcore run ab-test \ @@ -222,0 +249 @@ AgentCore CLI + --gateway csGateway \ @@ -224,2 +251,2 @@ AgentCore CLI - --control-endpoint control \ - --treatment-endpoint treatment \ + --control-target customer-support-control \ + --treatment-target customer-support-treatment \ @@ -229,4 +256 @@ AgentCore CLI - --treatment-weight 20 \ - --enable - - agentcore deploy + --treatment-weight 20 @@ -234 +258 @@ AgentCore CLI -This command adds the A/B test to your local `agentcore.json`. Run `agentcore deploy` to create it in your AWS account. The CLI also creates an AgentCore Gateway and registers both endpoints as targets. Optionally, use `--gateway <name>` to select an existing AgentCore Gateway that already has the endpoints set as targets. +The test is RUNNING as soon as the command returns. Pass `--disable-on-create` to create it stopped. The `--gateway` flag is required and must reference the gateway you deployed in Step 5. Only one test can be RUNNING per gateway at a time. The command prints the test’s job ID, which is also available from `--json` as the `id` field. You need this ID for the lifecycle commands below. @@ -343 +367 @@ AWS SDK (boto3) -## Step 6: Send traffic through the AgentCore Gateway +## Step 7: Send traffic through the AgentCore Gateway @@ -355 +379 @@ Session assignment is **sticky** : once a session ID is assigned to a variant, a -Save the following script as `loadgen.sh`, replacing `<gateway-id>` and `<target-name>` with the values from your deployment output: +Save the following script as `loadgen.sh`, replacing `<gateway-id>` and `<target-name>` with the values from your deployment output. You can also copy the full invocation URL from `agentcore view ab-test <ab-test-id>`: @@ -398 +422 @@ Run the script: -## Step 7: Get results +## Step 8: Get results @@ -407 +431 @@ AgentCore CLI -Get current results: +Get current results (replace `<ab-test-id>` with the job ID from Step 6): @@ -410 +434 @@ Get current results: - agentcore ab-test customerSupportTargetTest + agentcore view ab-test <ab-test-id> @@ -415 +439 @@ Get results as JSON: - agentcore ab-test customerSupportTargetTest --json + agentcore view ab-test <ab-test-id> --json @@ -485 +509 @@ The time it takes for results to appear depends primarily on the session timeout -## Step 8: Confirm results and stop the A/B test +## Step 9: Confirm results and stop the A/B test @@ -491 +515 @@ Once the A/B test reaches statistical significance, review the results and stop - 2. **Stop the A/B test.** Set `executionStatus` to `STOPPED`. Traffic routing ends immediately and all requests revert to the default target. See [Start, pause, and stop](./ab-testing-manage.html#manage-ab-test-start-stop). + 2. **Stop the A/B test.** Run `agentcore stop ab-test -i <ab-test-id>`. Traffic routing ends immediately and all requests revert to the default target. See [View, pause, resume, and stop](./ab-testing-manage.html#manage-ab-test-start-stop). @@ -496 +520 @@ Once the A/B test reaches statistical significance, review the results and stop -## Step 9: Deploy the winner +## Step 10: Deploy the winner @@ -501 +525 @@ After stopping the A/B test, route all traffic to the winning variant. - agentcore promote ab-test customerSupportTargetTest + agentcore promote ab-test -i <ab-test-id> @@ -504 +528 @@ After stopping the A/B test, route all traffic to the winning variant. -This stops the A/B test, updates the control endpoint to point to the winning version (for example, updating `control` from version 1 to version 2), and removes the treatment endpoint. Run `agentcore deploy` to apply the changes. +`promote` stops the A/B test (if still running), updates the control endpoint to point to the treatment version (for example, updating `control` from version 1 to version 2), and removes the treatment endpoint. Run `agentcore deploy` to apply the changes.