AWS devopsagent documentation change
Summary
Updated terminology from 'simulated verification' to 'automated verification testing', restructured setup steps into two capabilities (auto-trigger change review and automated verification testing), added network allowlist for verification environment, and removed detailed credential management instructions in favor of runtime IAM roles.
Security assessment
The change adds documentation about security features including network restrictions (allowlist) for the verification environment and IAM role-based access to internal services. This helps prevent data exfiltration and unauthorized access during automated testing. No specific vulnerability is mentioned.
Diff
diff --git a/devopsagent/latest/userguide/release-management-release-readiness-code-review.md b/devopsagent/latest/userguide/release-management-release-readiness-code-review.md index 268788bad..47a335c9a 100644 --- a//devopsagent/latest/userguide/release-management-release-readiness-code-review.md +++ b//devopsagent/latest/userguide/release-management-release-readiness-code-review.md @@ -7 +7 @@ -Getting startedPerforming a code reviewAutomated code reviewsSimulated verificatonReviewing code review resultsIntegrate with Kiro IDE and CLIIntegrate with Claude CodeIntegrate with AWS Transform customUsing code reviews in GitHubUsing code reviews in GitLabUsing code reviews in DevOps Agent chatAgentic safety guardrails +Getting startedPerforming a code reviewAutomated code reviewsAutomated verification testingReviewing code review resultsIntegrate with Kiro IDE and CLIIntegrate with Claude CodeIntegrate with AWS Transform customUsing code reviews in GitHubUsing code reviews in GitLabUsing code reviews in DevOps Agent chatAgentic safety guardrails @@ -11 +11 @@ Getting startedPerforming a code reviewAutomated code reviewsSimulated verificat -Release readiness code reviews evaluate your code changes for cross-repository dependency risks, internal standards compliance, and access-control correctness. It also performs simulated verification – it builds, runs, and tests your code changes – in a sandbox environment managed by the AWS DevOps Agent. +Release readiness code reviews evaluate your code changes for cross-repository dependency risks, internal standards compliance, and access-control correctness. It also performs automated verification testing – it builds, runs, and tests your code changes – in a verification environment managed by the AWS DevOps Agent. @@ -17 +17 @@ To use release readiness code reviews, complete the following setup steps. -### Step 1: Enable change reviews on your repositories +### Step 1: Enable capabilities on your repositories @@ -19 +19 @@ To use release readiness code reviews, complete the following setup steps. -Change reviews must be enabled on your connected GitHub or GitLab repositories before automated code reviews can trigger. +Code review and automated testing capabilities must be enabled on your connected GitHub or GitLab repositories before they can trigger. @@ -21 +21 @@ Change reviews must be enabled on your connected GitHub or GitLab repositories b -**For GitHub:** Navigate to the **Code Review and Automated Testing** section in your GitHub integration settings and verify the **Change review** checkbox is enabled for each repository. Change review is enabled by default when you connect repositories. For detailed instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-github.html). +The **Code Review and Automated Testing** section in your pipeline provider integration settings provides two per-repository capabilities: @@ -23 +23 @@ Change reviews must be enabled on your connected GitHub or GitLab repositories b -**For GitLab:** Navigate to the **Code Review and Automated Testing** section in your GitLab integration settings and enable change reviews for your projects. For detailed instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-gitlab.html). + * **Auto trigger change review** — When enabled, DevOps Agent automatically runs a release readiness code review each time a pull request or merge request is opened or updated. Review findings appear as inline comments on the PR/MR. @@ -25 +25 @@ Change reviews must be enabled on your connected GitHub or GitLab repositories b -### Step 2: Configure private VPC access for the simulated verification environment (optional) + * **Automated verification testing** — When enabled, DevOps Agent builds, runs, and tests your code changes in a managed verification environment during code reviews. This provides functional validation beyond static analysis. For more information, see Automated verification testing. @@ -27 +26,0 @@ Change reviews must be enabled on your connected GitHub or GitLab repositories b -Release readiness code reviews can perform simulated verification by building, running, and testing your code changes in a sandbox environment (see #Simulated-verificaton). If your code build process requires artifacts from internal systems — such as private image repositories (for example, Artifactory, Docker Hub Enterprise), internal build artifact stores, or dependent code repositories — you need to give the simulated verification environment access to a VPC that can reach those service endpoints. @@ -29 +27,0 @@ Release readiness code reviews can perform simulated verification by building, r -The simulated verification environment runs in an isolated sandbox with Internet access. By default, it does not have network access to your internal systems. To enable access, create a private connection and associate it with your pipeline provider ([GitHub](./connecting-to-cicd-pipelines-connecting-github.html) or [GitLab](./connecting-to-cicd-pipelines-connecting-gitlab.html)). The simulated verification environment uses the VPC associated with that private connection by creating and managing an ENI inside the VPC, giving the build sandbox network access to your internal services. @@ -31,26 +28,0 @@ The simulated verification environment runs in an isolated sandbox with Internet -###### Note - - __Internal services required during builds — such as package registries, container registries, and artifact stores — are typically located on the same private network as your version control and pipeline infrastructure. Associating the private connection with your pipeline provider ensures the simulated verification environment can reach these services through the same network path. - -###### Note - - __Integrating with VPCs in your account routes network traffic through your internal routes, following any network restrictions in place. - -To configure private VPC access for simulated verification: - - 1. Create a private connection that targets the VPC where your internal build services are reachable. For instructions, see [Connecting to privately hosted tools](./configuring-integrations-and-knowledge-connecting-to-privately-hosted-tools.html). - - 2. Open the AWS DevOps Agent console and navigate to your Agent Space. - - 3. Go to the **Capabilities** tab and select your pipeline provider (GitHub or GitLab). - - 4. In the **Code Review and Automated Testing** section, associate the private connection with your pipeline provider by selecting it from the available connections. - - 5. For the **Runtime role** , select an IAM role that DevOps Agent will assume when accessing internal services during builds. This role must have permission to access AWS Secrets Manager in the same AWS account as your Agent Space. We recommend using a different role from your primary agent role. - - 6. Choose **Save** to apply your configuration. - - - - -Once associated, the simulated verification environment will provision an ENI in the private connection's VPC, giving it direct network access to your internal services during code review builds. @@ -58 +30 @@ Once associated, the simulated verification environment will provision an ENI in -### Step 3: Add credentials for simulated verification (optional) +You can enable or disable each capability independently per repository, allowing you to use change reviews without verification testing or vice versa. @@ -60 +32 @@ Once associated, the simulated verification environment will provision an ENI in -When the simulated verification environment builds your code project, it may need credentials to authenticate with internal services — for example, credentials for Artifactory, Docker Hub, private npm registries, or internal Maven repositories. +The section also includes: @@ -62 +34 @@ When the simulated verification environment builds your code project, it may nee -These credentials are stored in AWS Secrets Manager in the same AWS account as your Agent Space and are securely injected into the simulated verification environment at build time. + * **Runtime role** (optional) — Choose the IAM role that DevOps Agent assumes to run automated capabilities on your selected repositories. This role is used when accessing internal services during builds, such as private package registries or artifact stores. For more information, see Step 2. @@ -64 +35,0 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y -#### Storing credentials in Secrets Manager @@ -66 +36,0 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - 1. Open the [AWS Secrets Manager console](https://console.aws.amazon.com/secretsmanager/) in the same account and Region as your Agent Space. @@ -68 +37,0 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - 2. Choose **Store a new secret**. @@ -70 +39 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - 3. Select **Other type of secret**. +**For GitHub:** Navigate to the **Code Review and Automated Testing** section in your GitHub integration settings and enable the capabilities for each repository. Both capabilities are enabled by default when you connect repositories. For detailed instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-github.html). @@ -72 +41 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - 4. Enter the credential as a key/value pair. For example: +**For GitLab:** Navigate to the **Code Review and Automated Testing** section in your GitLab integration settings and enable the capabilities for your projects. For detailed instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-gitlab.html). @@ -74 +43 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - * Key: `ARTIFACTORY_TOKEN`, Value: `<your-token>` +### Step 2: Configure private VPC access for the verification testing environment (optional) @@ -76 +45 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - * Key: `DOCKER_USERNAME`, Value: `<your-username>` +Release readiness code reviews can perform automated verification testing by building, running, and testing your code changes in a verification environment (see Automated verification testing). If your code build process requires artifacts from internal systems — such as private image repositories (for example, Artifactory, Docker Hub Enterprise), internal build artifact stores, or dependent code repositories — you need to give the verification testing environment access to a VPC that can reach those service endpoints. @@ -78 +47 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - * Key: `DOCKER_PASSWORD`, Value: `<your-password>` +By default, the verification testing environment does not have network access to your internal systems. To enable access, create a private connection and associate it with your pipeline provider ([GitHub](./connecting-to-cicd-pipelines-connecting-github.html) or [GitLab](./connecting-to-cicd-pipelines-connecting-gitlab.html)). The verification testing environment uses the VPC associated with that private connection by creating and managing an ENI inside the VPC, giving the build environment network access to your internal services. @@ -80,10 +49 @@ These credentials are stored in AWS Secrets Manager in the same AWS account as y - 5. Name the secret with a descriptive prefix, such as `devops-agent/build/artifactory` or `devops-agent/build/docker-hub`. - - 6. Complete the secret creation. - - - - -#### Mapping credentials to the simulated verification environment - -After storing your credentials in Secrets Manager, use a [DevOps Agent Skills](./about-aws-devops-agent-devops-agent-skills.html) to map each secret ARN to the intended technology and instruct the agent on how to populate credential configurations in the sandbox environment. +###### Note @@ -91 +51 @@ After storing your credentials in Secrets Manager, use a [DevOps Agent Skills](. -Create a Skill with instructions that tell the agent which secrets to retrieve and how to use them. For example: + __Integrating with VPCs in your account routes network traffic through your internal routes, following any network restrictions in place. @@ -92,0 +53 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a +To configure private VPC access for verification testing: @@ -94,6 +55 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - --- - name: simulated-verification-credentials - description: Instructions for configuring build credentials during simulated - verification. Use this skill when setting up the sandbox environment for - code review builds. - --- + 1. Create a private connection that targets the VPC where your internal build services are reachable. For instructions, see [Connecting to privately hosted tools](./configuring-integrations-and-knowledge-connecting-to-privately-hosted-tools.html). @@ -101 +57 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - # Build Credential Configuration + 2. Open the AWS DevOps Agent console and navigate to your Agent Space. @@ -103,2 +59 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - When setting up the simulated verification environment, retrieve and configure - the following credentials: + 3. Go to the **Capabilities** tab and select your pipeline provider (GitHub or GitLab). @@ -106 +61 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - ## Docker Hub + 4. In the **Code Review and Automated Testing** section, associate the private connection with your pipeline provider by selecting it from the available connections. @@ -108,4 +63 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - - Secret ARN: `arn:aws:secretsmanager:us-east-1:123456789012:secret:devops-agent/build/docker-hub` - - Technology: Docker - - Set the `DOCKER_USERNAME` and `DOCKER_PASSWORD` environment variables from this secret - - Run `docker login` with these credentials before executing builds + 5. For the **Runtime role** , select an IAM role that DevOps Agent will assume when accessing internal services during builds. This role must have permission to access AWS Secrets Manager in the same AWS account as your Agent Space. We recommend using a different role from your primary agent role. @@ -113 +65 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - ## Artifactory + 6. Choose **Save** to apply your configuration. @@ -115,3 +66,0 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - - Secret ARN: `arn:aws:secretsmanager:us-east-1:123456789012:secret:devops-agent/build/artifactory` - - Technology: Artifactory (npm registry) - - Write the token to `.npmrc` as `//your-artifactory.example.com/:_authToken=<token>` @@ -119 +67,0 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - ## Application Secrets @@ -121,5 +68,0 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a - - Secret ARN: `arn:aws:secretsmanager:us-east-1:123456789012:secret:devops-agent/build/app-env` - - Create a `.env` file in the project root with the following keys from this secret: - - `DATABASE_URL` - - `API_KEY` - - `REDIS_URL` @@ -127 +70 @@ Create a Skill with instructions that tell the agent which secrets to retrieve a -The agent automatically loads this Skill during simulated verification and uses the instructions to retrieve secrets from Secrets Manager and configure the sandbox environment — setting environment variables, writing `.env` files, or populating tool-specific configuration files as needed. +Once associated, the verification testing environment will provision an ENI in the private connection's VPC, giving it direct network access to your internal services during code review builds. @@ -180 +123,9 @@ Findings appear as inline comments on the affected lines of code, with the overa -## Simulated verificaton +## Automated verification testing + +When a release readiness risk assessment is triggered, DevOps Agent creates an AWS-managed verification environment and clones your code into it. The environment runs on dedicated compute resources with network restrictions that limit access to trusted services for build, artifact storage, and retrieval. + +DevOps Agent reads your application's code and project files to determine the required build tools and dependencies, then installs them in the verification environment. After successfully building your application, the agent generates a testing plan and runs it to identify functional risks — such as edge cases that might result in failures or unexpected behavior. + +Findings from verification testing are included in the final release readiness report alongside standards, dependency, and access-control findings. + +You can use [Agent instructions](./about-aws-devops-agent-agent-instructions.html) (AGENTS.md) to tune how verification testing is performed — for example, specifying which test commands to run, what constitutes a passing build, or which parts of the application to exercise during verification. @@ -182 +133 @@ Findings appear as inline comments on the affected lines of code, with the overa -When a release readiness review performed, the agent creates a sandboxed environment to build, run, and test your application for simulated verification. This provides functional validation beyond static code analysis — the agent compiles your code, starts the application in an isolated sandbox, and executes tests to verify that the changes work as expected. +### Allowed network destinations @@ -184 +135 @@ When a release readiness review performed, the agent creates a sandboxed environ -Findings from sandboxed functional testing are included in the final release readiness report alongside standards, dependency, and access-control findings. +The verification testing environment has outbound network access restricted to a predefined allowlist. Your application can reach the following domains during validation: @@ -186 +137,24 @@ Findings from sandboxed functional testing are included in the final release rea -You can use [Agent instructions](./about-aws-devops-agent-agent-instructions.html) (AGENTS.md) to tune how simulated validation testing is performed — for example, specifying which test commands to run, what constitutes a passing build, or which parts of the application to exercise during verification. +Domain | Purpose +---|--- +`.amazonaws.com`, `.aws.amazon.com` | AWS services +`.public.ecr.aws` | Amazon ECR Public +`.docker.com`, `.docker.io` | Docker Hub +`.github.com`, `.githubusercontent.com` | GitHub +`.gitlab.com` | GitLab +`.npmjs.com`, `.npmjs.org` | npm registry +`.pypi.org`, `.pypi.python.org`, `.pythonhosted.org` | Python Package Index +`.crates.io`, `.rustup.rs` | Rust packages +`.maven.org`, `.gradle.org` | Java/Gradle packages +`.nuget.org` | .NET packages +`.rubygems.org`, `.ruby-lang.org` | Ruby packages +`.golang.org`, `.pkg.go.dev`, `.goproxy.io` | Go packages +`.nodejs.org`, `.yarnpkg.com` | Node.js +`.alpinelinux.org`, `.debian.org`, `.ubuntu.com`, `.centos.org`, `.fedoraproject.org` | Linux distribution repositories +`.cloudfront.net` | CloudFront distributions +`.google.com`, `.googleapis.com` | Google APIs +`.microsoft.com`, `.visualstudio.com` | Microsoft services +`.sourceforge.net`, `.bitbucket.org` | Source hosting + +###### Note + + __If your application requires network access to domains not on this list, you can connect your verification testing environment to a VPC will cause the agent to use your own network firewall settings, allowing you to configure access to any of the services your application requires. @@ -190 +164 @@ You can use [Agent instructions](./about-aws-devops-agent-agent-instructions.htm -Each code review produces a report accessible in the Release Manager section of the DevOps Agent web app. Reports include: +Each code review produces a report accessible in the **Changes** page of the DevOps Agent web app. Reports include: @@ -248 +222 @@ To use release readiness code reviews in [AWS Transform custom](https://docs.aws -Prerequisites: GitHub repository connected to your Agent Space with automated reviews enabled. +Prerequisites: GitHub repository connected to your Agent Space with automated reviews enabled. For setup instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-github.html). @@ -261 +235 @@ Prerequisites: GitHub repository connected to your Agent Space with automated re -Prerequisites: GitLab repository connected to your Agent Space with automated reviews enabled. +Prerequisites: GitLab project connected to your Agent Space with automated reviews enabled. For setup instructions, see [Configuring Code Review and Automated Testing](./connecting-to-cicd-pipelines-connecting-gitlab.html). @@ -306,0 +281,4 @@ Read-only operations such as `describe_*`, `get_*`, and `list_*` are permitted. +### Sequential phase enforcement +