AWS wellarchitected: Rewrite ADVREL04 reliability topic to failure and recovery
Summary
Replaces change-management guidance with failure, recovery, and capacity management content covering DR runbooks, partner outages, defensive architecture, backup/RTO-RPO, and privacy-aware data replication.
Security assessment
The added guidance recommends private connectivity options (Direct Connect/PrivateLink) for critical partner integrations and also flags cross-region backup replication against privacy regulations, making it security-relevant documentation. No specific vulnerability or incident is referenced.
Evidence
+ * Relying on internet-based connections for critical partner integrations without [AWS Direct Connect](https://aws.amazon.com/directconnect/) or [AWS PrivateLink](https://aws.amazon.com/privatelink/) alternatives.
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel04.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel04.md index 23edb3e..1000574 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advrel04.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel04.md @@ -7 +7 @@ -# Change management +Capability intentCommon issues to watch for @@ -9 +9,5 @@ -ADVREL04: How do you prevent regression from changes in your application and infrastructure? +# Failure, recovery, and capacity management + +Advertising systems operate at the intersection of owned infrastructure and external partner dependencies (DSPs, SSPs, measurement providers, and CDN origins). A failure in any layer, whether an internal component, an external partner, or a data store, translates directly into lost auction revenue or measurement gaps. This capability addresses how you detect, contain, and recover from failures across all three dimensions: proactive DR readiness, defensive architecture against external failures, and data recovery within regulatory constraints. + +ADVREL04: How do you manage failures, maintain availability during partner outages, and recover data within your recovery objectives? @@ -10,0 +15,26 @@ ADVREL04: How do you prevent regression from changes in your application and inf +[ADVREL04-BP01 Create disaster recovery (DR) runbooks, and regularly test documented backup and restoration processes](./advrel04-bp01.html) +[ADVREL04-BP02 Architect defensively against failures](./advrel04-bp02.html) +[ADVREL04-BP03 Monitor and manage partner SLAs against actuals](./advrel04-bp03.html) +[ADVREL04-BP04 Design your workloads to withstand failures of individual components](./advrel04-bp04.html) +[ADVREL04-BP05 Back up data in multiple locations with consideration for regulatory requirements](./advrel04-bp05.html) + +## Capability intent + + * Disaster recovery runbooks are documented, regularly tested with SSP/DSP partner coordination, and validated against realistic failure conditions using [AWS Fault Injection Service](https://aws.amazon.com/fis/). + + * Your advertising system continues operating when external partners experience failures, using circuit breakers, retries, and fallbacks that help avoid cascading impact across the auction pipeline. + + * Partner SLAs are monitored against actuals with [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) custom metrics, and SLA degradation triggers automated protective actions before revenue impact. + + * Individual component failures (compute, databases, and caches) are absorbed by cell-based architectures with distributed resources across Availability Zones. + + * Data is backed up in multiple locations with backup strategies aligned to per-tier RTO/RPO objectives and with consideration for privacy regulations governing advertising audience data. + + + + +## Common issues to watch for + + * Never testing backup restoration, discovering that backups are corrupt or incomplete during an actual disaster. + + * DR runbooks that describe the desired end-state but not the step-by-step procedure, making them unusable under pressure. @@ -12 +42 @@ ADVREL04: How do you prevent regression from changes in your application and inf -Changes to your advertising workload or its environment must be anticipated and accommodated to achieve reliable operation of the workload. Changes include those imposed on your workload such as spikes in demand, as well as those from within such as feature deployments and security patches. + * External partner failures cascading into auction failures because of missing circuit breakers or fallback behavior. @@ -14 +44 @@ Changes to your advertising workload or its environment must be anticipated and -Maintain reliability during application and infrastructure changes. Implement comprehensive testing (like regression, performance, and canary) in CI/CD pipelines to monitor impact on critical metrics. Additionally, use phased deployment strategies (like blue/green and rolling) to minimize service disruption and quickly recover from issues. + * Relying on internet-based connections for critical partner integrations without [AWS Direct Connect](https://aws.amazon.com/directconnect/) or [AWS PrivateLink](https://aws.amazon.com/privatelink/) alternatives. @@ -16 +46 @@ Maintain reliability during application and infrastructure changes. Implement co -###### Best practices + * No fallback behavior when external data providers (DMPs, audience platforms) are unavailable, so bids proceed with no targeting data or are abandoned entirely. @@ -18 +48 @@ Maintain reliability during application and infrastructure changes. Implement co - * [ADVREL04-BP01 Through your CI/CD pipeline, employ end-to-end regression, performance, and canary testing](./advrel04-bp01.html) + * Not considering privacy regulations when planning cross-region data replication for backup. EU user data may not be replicated to non-EU regions. @@ -20 +50 @@ Maintain reliability during application and infrastructure changes. Implement co - * [ADVREL04-BP02 Deploy new code or resources in staggered phases, separated by sufficient time, to verify that the changes are successful](./advrel04-bp02.html) + * Over-provisioning backup solutions beyond what RPO/RTO requirements justify, inflating costs without proportional recovery benefit. @@ -31 +61 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -ADVREL03-BP04 Reserve appropriate capacity of services in the supported Regions +ADVREL03-BP04 Reserve appropriate capacity of services in the supported regions @@ -33 +63 @@ ADVREL03-BP04 Reserve appropriate capacity of services in the supported Regions -ADVREL04-BP01 Through your CI/CD pipeline, employ end-to-end regression, performance, and canary testing +ADVREL04-BP01 Create disaster recovery (DR) runbooks, and regularly test documented backup and restoration processes