AWS Security ChangesHomeSearch

AWS wellarchitected: Rewrite ADVREL04-BP02 to defensive architecture against failures

Service: wellarchitected · 2026-11-11 · Documentation low

File: wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp02.md · Type: network

Summary

Replaces the previous blue/green and rolling deployment guidance with a new resilience-focused best practice: circuit breakers, retries, and fallbacks for external partner dependencies, static stability caching of partner configuration, Direct Connect/PrivateLink for high-volume partners, and quarterly fault-injection validation. Also updates related links, services, and the adjacent best-practice titles.

Security assessment

The rewrite is primarily availability/reliability guidance (circuit breakers, static stability, fallback validation) rather than a fix for a specific vulnerability or incident. The only security-adjacent element is the recommendation to use AWS PrivateLink/Direct Connect instead of the public internet for high-volume partner traffic, which reduces public network exposure and latency variance, but it is framed as a latency/stability concern, not a security control, and no CVE or weakness is referenced.

Evidence

+  3. **Establish Direct Connect or PrivateLink for highest-volume partner connections:** For partners handling more than 10% of your bid volume, establish AWS PrivateLink or Direct Connect for stable, private connectivity. Monitor latency variance on these connections. Internet-based connections show significantly higher p99 latency variability than private connections.

Diff

diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp02.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp02.md
index 34b9133..921ca19 100644
--- a//wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp02.md
+++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp02.md
@@ -7 +7 @@
-Implementation guidanceKey AWS servicesResources
+Implementation guidanceResources
@@ -9 +9 @@ Implementation guidanceKey AWS servicesResources
-# ADVREL04-BP02 Deploy new code or resources in staggered phases, separated by sufficient time, to verify that the changes are successful
+# ADVREL04-BP02 Architect defensively against failures
@@ -11 +11,38 @@ Implementation guidanceKey AWS servicesResources
-Implement gradual, phased deployments to minimize risks and service impacts when updating systems. 
+Some system failures come from external partners or dependencies that are outside of your control. Implement circuit breakers, retries, and fallbacks to gracefully handle failures from external dependencies, and avoid cascading failures within your system. 
+
+**Desired outcome:**
+
+  * External dependency failures are isolated and don't cascade into auction failures. 
+
+  * Your system continues to serve bids with graceful degradation (default targeting and cached data) when external services are unavailable. 
+
+  * Direct Connect or PrivateLink provides stable, low-latency connectivity for critical partner integrations. 
+
+
+
+
+**Common anti-patterns:**
+
+  * Treating external partner dependencies as consistently available, with no circuit breaker, retry, or fallback. 
+
+  * Making control plane calls (e.g., describing available DSPs) on every bid request instead of caching pre-computed state. 
+
+  * Routing high-volume partner traffic over the public internet, accepting high p99 latency variance for critical integrations. 
+
+  * Never exercising fallback paths, so degraded-mode behavior is unverified until a real partner outage. 
+
+
+
+
+**Benefits of establishing this best practice:**
+
+  * External dependency failures are isolated and don't cascade into auction failures. 
+
+  * The system keeps serving bids with graceful degradation (default targeting and cached data) when partners are unavailable. 
+
+  * Private connectivity (Direct Connect or PrivateLink) gives stable, low-latency links for critical partner integrations. 
+
+
+
+
+**Level of risk exposed if this best practice is not established:** High 
@@ -15 +52,5 @@ Implement gradual, phased deployments to minimize risks and service impacts when
-When deploying new code or resources, it is possible for unintended results to occur. Various deployment strategies can be used to reduce frequency and service impact. 
+Advertising systems depend on external partners: DSPs submitting bids, DMPs supplying audience segments, and measurement providers validating impressions. Each partner is a failure domain you don't control. Outages, latency spikes, and error bursts happen without warning during your auction windows. Every synchronous call to an external dependency couples your auction result to that partner's availability. A single partner outage can then cascade into a system-wide auction failure. 
+
+Static stability keeps the data plane running when the control plane is impaired. Bid requests that must complete in single-digit milliseconds cannot depend on control plane freshness. If the system asks which partners are active on every request, the control plane is in the critical path of the data plane. Any control plane delay then becomes an auction delay. Refresh configuration on a different cadence than you consume it to insulate the data plane from upstream instability. 
+
+A fallback path that has never executed under production traffic is unproven. Confirm graceful degradation by exercising failure modes under realistic conditions. When you treat resilience validation as a periodic discipline, you can discover systemic brittleness earlier and more efficiently. 
@@ -17 +58 @@ When deploying new code or resources, it is possible for unintended results to o
-By making changes through a blue/green deployment methodology, you can significantly reduce the impact of any potential issues and avoid downtime. 
+### Implementation steps
@@ -19 +60 @@ By making changes through a blue/green deployment methodology, you can significa
-When a blue/green deployment isn't possible, a rolling deployment methodology should be used to reduce the number of resources being modified simultaneously. With a rolling deployment, changes are made in small batches, with a pre-determined amount of buffer time between batches. If an issue occurs with the deployment, the unchanged resources can continue handling traffic, avoiding downtime. 
+  1. **Implement circuit breakers for all external partner connections:** Deploy Resilience4j or equivalent circuit breakers on all DSP, SSP, DMP, and measurement provider connections. Set open threshold at 50% failure rate over a 30-second window. Configure fallback behavior for each connection: DSP unavailable (skip DSP from auction), DMP unavailable (use cached audience segments), measurement provider unavailable (record locally, sync later). 
@@ -21 +62 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-## Key AWS services
+  2. **Use Static Stability patterns for partner configuration:** Pre-fetch and cache partner configurations (active DSPs, bid floors, and targeting parameters) at service startup and refresh on a configurable interval (every 5 minutes). During a control plane failure, the service continues operating with the last-known-good configuration rather than blocking on configuration retrieval. 
@@ -23 +64 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-  * [AWS CloudFormation](https://aws.amazon.com/cloudformation/)
+  3. **Establish Direct Connect or PrivateLink for highest-volume partner connections:** For partners handling more than 10% of your bid volume, establish AWS PrivateLink or Direct Connect for stable, private connectivity. Monitor latency variance on these connections. Internet-based connections show significantly higher p99 latency variability than private connections. 
@@ -25 +66 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-  * [Amazon Elastic Container Service (ECS)](https://aws.amazon.com/ecs/)
+  4. **Validate fallback paths with regular fault injection:** Quarterly, inject failures into each external dependency category and verify that: the correct circuit breaker activates, fallback behavior produces valid (if degraded) auction results, and the system recovers automatically when the dependency recovers. 
@@ -32 +73,23 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-  * [Blue/Green Deployments on AWS](https://docs.aws.amazon.com/whitepapers/latest/blue-green-deployments/welcome.html)
+**Related best practices:**
+
+  * [ADVREL02-BP01 To allow fast and graceful failure of latency-sensitive services, avoid exponential backoff and retry](./advrel02-bp01.html)
+
+  * [ADVREL04-BP04 Design your workloads to withstand failures of individual components](./advrel04-bp04.html)
+
+  * [ADVREL04-BP03 Monitor and manage partner SLAs against actuals](./advrel04-bp03.html)
+
+
+
+
+**Related documents:**
+
+  * [Static stability using Availability Zones](https://aws.amazon.com/builders-library/static-stability-using-availability-zones/)
+
+  * [Reducing the Scope of Impact with Cell-Based Architecture](https://docs.aws.amazon.com/wellarchitected/latest/reducing-scope-of-impact-with-cell-based-architecture/reducing-scope-of-impact-with-cell-based-architecture.html)
+
+
+
+
+**Related services:**
+
+  * [AWS Direct Connect](https://aws.amazon.com/directconnect/)
@@ -34 +97 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-  * [Rolling deployments](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/rolling-deployments.html)
+  * [Amazon DynamoDB](https://aws.amazon.com/dynamodb/)
@@ -36 +99 @@ When a blue/green deployment isn't possible, a rolling deployment methodology sh
-  * [Deployment methods](Users/jblatch/Downloads/%E2%80%A2%20https:/docs.aws.amazon.com/whitepapers/latest/practicing-continuous-integration-continuous-delivery/deployment-methods.html)
+  * [Amazon ElastiCache](https://aws.amazon.com/elasticache/)
@@ -47 +110 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please
-ADVREL04-BP01 Through your CI/CD pipeline, employ end-to-end regression, performance, and canary testing
+ADVREL04-BP01 Create disaster recovery (DR) runbooks, and regularly test documented backup and restoration processes
@@ -49 +112 @@ ADVREL04-BP01 Through your CI/CD pipeline, employ end-to-end regression, perform
-Failure management 
+ADVREL04-BP03 Monitor and manage partner SLAs against actuals