AWS prescriptive-guidance medium security documentation change
Summary
Restructured testing methodology sections (swapped emulation/cloud testing order), added details about security monitoring gaps in mock testing, expanded limitations of emulators, and added enterprise security policy considerations for cloud testing.
Security assessment
The change explicitly highlights that mock frameworks cannot detect IAM policy violations nor trigger security services like GuardDuty/CloudTrail. This directly documents security-relevant gaps in testing methodologies that could lead to undetected policy misconfigurations or missing security monitoring in production environments.
Diff
diff --git a/prescriptive-guidance/latest/serverless-application-testing/techniques.md b/prescriptive-guidance/latest/serverless-application-testing/techniques.md index af4518855..d5c6a0eea 100644 --- a//prescriptive-guidance/latest/serverless-application-testing/techniques.md +++ b//prescriptive-guidance/latest/serverless-application-testing/techniques.md @@ -5 +5 @@ -Mock testingTesting in the cloudEmulation testing +Mock testingEmulation testingTesting in the cloud @@ -9 +9 @@ Mock testingTesting in the cloudEmulation testing -There are three main approaches to running tests against serverless application code: mock testing, testing in the cloud, and emulation testing. You can generally find a mix of these approaches in use within the scope of a single project. For example, you might use mock testing when you’re developing your code locally, but your code might be tested in the cloud as part of a nightly continuous integration and continuous delivery (CI/CD) process. +There are three main approaches to running tests against serverless application code: mock testing, emulation testing, and testing in the cloud. You can generally find a mix of these approaches in use within the scope of a single project. For example, you might use mock testing when you're developing your code locally, emulation testing to more closely replicate service behavior before deployment, and cloud testing as part of a nightly continuous integration and continuous delivery (CI/CD) process. @@ -13 +13 @@ There are three main approaches to running tests against serverless application -Mock testing is a strategy where you create replacement objects in your code that simulate the behavior of cloud services. For example, you can write a test that uses a mock of the Amazon S3 service, and you can configure that mock to return a specific response object whenever the **CreateObject** method is called. When the test runs, the mock returns the response without calling Amazon S3 or any other service endpoints. +Mock testing is a strategy where you create replacement objects in your code that simulate the behavior of cloud services. For example, you can write a test that uses a mock of the Amazon S3 service, and you can configure that mock to return a specific response object whenever the `PutObject` method is called. When the test runs, the mock returns the response without calling Amazon S3 or any other service endpoints. @@ -25 +25 @@ The advantages of using mocks include the following: - * Like emulators, mock frameworks can provide fast local development after they’ve been configured. + * Like emulators, mock frameworks can provide fast local development after they've been configured. @@ -44 +44 @@ Mock testing has these disadvantages: - * Mock tests might pass in desktop environments but fail in the cloud. + * Mock tests might pass locally but fail in the cloud because they simulate — rather than replicate — the behavior of real services, leaving environment-specific issues undetected. @@ -46 +46 @@ Mock testing has these disadvantages: - * Mock frameworks, like emulators, are limited in testing or detecting AWS Identity and Access Management (IAM) policy or quota limitations. + * Mock frameworks, like emulators, cannot detect AWS Identity and Access Management (IAM) policy violations, service quota limits, or payload size constraints, nor do they trigger ancillary services such as Amazon CloudWatch, AWS CloudTrail, or Amazon GuardDuty that may affect application behavior in a deployed environment. @@ -48 +48 @@ Mock testing has these disadvantages: - * Although mocks are better at simulating what an application will do when authorization fails or a quota is exceeded, mock testing can’t determine which outcome will actually occur when the code is deployed to the production environment. + * Although mocks are better at simulating what an application will do when authorization fails or a quota is exceeded, mock testing can't determine which outcome will actually occur when the code is deployed to the production environment. @@ -53 +53 @@ Mock testing has these disadvantages: -## Testing in the cloud +## Emulation testing @@ -55 +55 @@ Mock testing has these disadvantages: -Testing in the cloud is the process of running tests against code that is deployed to a cloud environment instead of a desktop environment. The value of testing in the cloud increases with cloud-native application development. For example: +Emulation testing is enabled by locally running applications known as _emulators_ that resemble AWS services. Emulators have APIs that are similar to their cloud counterparts and provide similar return values. They can also simulate state changes that are initiated by API calls. For example, an Amazon S3 emulator might store an object on the local disk when the `PutObject` method is called. When `GetObject` is called with the same key, the emulator reads the same object from disk and returns it. @@ -57 +57 @@ Testing in the cloud is the process of running tests against code that is deploy - * You can run tests in the cloud against the most recent APIs and return values. +The advantages of emulation testing include the following: @@ -59 +59 @@ Testing in the cloud is the process of running tests against code that is deploy - * Your tests can cover IAM policies, service quotas, configurations, and all services. + * It provides the most familiar environment for developers who are used to developing code exclusively in a local environment. For example, if you're familiar with the development of an _n_ -tier application, you might have a database engine and web server, similar to those running in production, running on your local machine to provide quick, local, isolated test capability. @@ -61 +61 @@ Testing in the cloud is the process of running tests against code that is deploy - * Your cloud test environment most closely resembles your production runtime environment, so tests that you run in the cloud are likely to achieve the most consistent results as they progress through environments. + * It doesn't require any changes to cloud infrastructure (such as developer cloud accounts), so it's easy to implement with existing testing patterns. Emulation testing provides the benefits of fast, local development iterations. @@ -66 +66 @@ Testing in the cloud is the process of running tests against code that is deploy -Drawbacks to testing in the cloud include the following: +However, emulators have several drawbacks: @@ -68 +68 @@ Drawbacks to testing in the cloud include the following: - * Deployments to cloud environments typically take more time than deployments to desktop environments. Tools such as [AWS Serverless Application Model (AWS SAM) Accelerate](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/accelerate.html), [AWS Cloud Development Kit (AWS CDK) watch mode](https://docs.aws.amazon.com/cdk/v2/guide/cli.html#cli-deploy-watch), and [SST](https://sst.dev/) help decrease the latency involved with cloud deployment iterations. + * They're often hard to set up and replicate, especially when they're used as a part of CI/CD pipelines. This might increase the workload and maintenance for IT staff or for developers who manage their own software installations. @@ -70 +70 @@ Drawbacks to testing in the cloud include the following: - * Cloud testing can incur additional service costs, unlike local testing, which uses your local development environment. + * Emulated features and APIs typically lag behind services changes and might hinder the adoption of new features until support is added. @@ -72 +72 @@ Drawbacks to testing in the cloud include the following: - * Testing in the cloud might be less feasible if you don’t have high-speed internet access. + * Emulators might require support, updates, bug fixes, or feature parity enhancements, which are the responsibility of the emulator author (which is often a third-party company). @@ -74 +74 @@ Drawbacks to testing in the cloud include the following: - * Testing in the cloud typically requires cloud environments that are isolated from one another. Environment boundaries are often drawn at the stack level in shared accounts for developer environments, sometimes by using namespace type strategies such as using prefixes to identify ownership. For pre-production and production environments, boundaries are typically drawn at the account level to insulate workloads from noisy neighbor problems, to support least privilege security controls, and to protect sensitive data. The requirement to create isolated environments might place additional burdens on DevOps teams, especially if they’re in an enterprise that has strict controls around accounts and infrastructure. + * Tests that rely on emulators can provide successful results locally, but they might fail in the cloud due to interactions with other aspects of AWS such as IAM policies and quotas, which are generally not emulated. @@ -75,0 +76 @@ Drawbacks to testing in the cloud include the following: + * Some AWS services don't have emulators available, so if you rely only on emulation, you might not have a satisfactory testing option for portions of your application. @@ -79 +79,0 @@ Drawbacks to testing in the cloud include the following: -## Emulation testing @@ -81 +81 @@ Drawbacks to testing in the cloud include the following: -Emulation testing is enabled by locally running applications known as _emulators_ that resemble AWS services. Emulators have APIs that are similar to their cloud counterparts and provide similar return values. They can also simulate state changes that are initiated by API calls. For example, an Amazon S3 emulator might store an object on the local disk when the **PutObject** method is called. When **GetObject** is called with the same key, the emulator reads the same object from disk and returns it. +## Testing in the cloud @@ -83 +83 @@ Emulation testing is enabled by locally running applications known as _emulators -The advantages of emulation testing include the following: +Testing in the cloud is the process of running tests against code that is deployed to a cloud environment instead of a desktop environment. The value of testing in the cloud increases with cloud-native application development. For example: @@ -85 +85 @@ The advantages of emulation testing include the following: - * It provides the most familiar environment for developers who are used to developing code exclusively in a local environment. For example, if you’re familiar with the development of an _n_ -tier application, you might have a database engine and web server, similar to those running in production, running on your local machine to provide quick, local, isolated test capability. + * You can run tests in the cloud against the most recent APIs and service features, ensuring your tests reflect the latest return values and behaviors as AWS continues to launch new services and capabilities. @@ -87 +87 @@ The advantages of emulation testing include the following: - * It doesn’t require any changes to cloud infrastructure (such as developer cloud accounts), so it’s easy to implement with existing testing patterns. Emulation testing provides the benefits of fast, local development iterations. + * Your tests can cover IAM policies, service quotas, configurations, and all services. @@ -88,0 +89 @@ The advantages of emulation testing include the following: + * Your cloud test environment most closely resembles your production runtime environment, so tests that you run in the cloud are likely to achieve the most consistent results as they progress through environments. @@ -92 +92,0 @@ The advantages of emulation testing include the following: -However, emulators have several drawbacks: @@ -94 +94 @@ However, emulators have several drawbacks: - * They’re often hard to set up and replicate, especially when they’re used as a part of CI/CD pipelines. This might increase the workload and maintenance for IT staff or for developers who manage their own software installations. +Drawbacks to testing in the cloud include the following: @@ -96 +96 @@ However, emulators have several drawbacks: - * Emulated features and APIs typically lag behind services changes and might hinder the adoption of new features until support is added. + * Deployments to cloud environments typically take more time than deployments to desktop environments. Tools such as [AWS Serverless Application Model (AWS SAM) Accelerate](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/accelerate.html), [AWS Cloud Development Kit (AWS CDK) watch mode](https://docs.aws.amazon.com/cdk/v2/guide/cli.html#cli-deploy-watch), and [SST](https://sst.dev/) help decrease the latency involved with cloud deployment iterations. @@ -98 +98 @@ However, emulators have several drawbacks: - * Emulators might require support, updates, bug fixes, or feature parity enhancements, which are the responsibility of the emulator author (which is often a third-party company). + * Cloud testing can incur additional service costs, unlike local testing, which uses your local development environment. @@ -100 +100,3 @@ However, emulators have several drawbacks: - * Tests that rely on emulators can provide successful results locally, but they might fail in the cloud due to interactions with other aspects of AWS such as IAM policies and quotas, which are generally not emulated. + * Testing in the cloud might be less feasible if you don't have high-speed internet access. + + * In regulated industries, enterprise security policies may restrict developer access to cloud environments, making it difficult or impossible to run cloud tests as part of a local development workflow. @@ -102 +104 @@ However, emulators have several drawbacks: - * Some AWS services don’t have emulators available, so if you rely only on emulation, you might not have a satisfactory testing option for portions of your application. + * Environment boundaries are often drawn at the stack level in shared accounts for developer environments, sometimes by using namespace type strategies such as using prefixes to identify ownership. For pre-production and production environments, boundaries are typically drawn at the account level to insulate workloads from noisy neighbor problems, to support least privilege security controls, and to protect sensitive data. The requirement to create isolated environments might place additional burdens on DevOps teams, especially if they're in an enterprise that has strict controls around accounts and infrastructure. @@ -115 +117 @@ Introduction -Challenges +Challenges when testing serverless applications