AWS wellarchitected: ADVREL04-BP01 rewritten: DR runbooks and backup restoration testing
Summary
Replaces the CI/CD regression/canary testing best practice with a disaster recovery best practice: documenting DR runbooks per Tier 1 component, quarterly backup restoration validation, automated backup monitoring (CloudWatch alarms for failed RDS backups, disabled DynamoDB PITR, S3 lifecycle failures), and annual partner-coordinated regional failover exercises. Adds desired outcomes, anti-patterns, implementation steps, related best practices/documents, and updates the section label to 'Failure, recovery, and capacity management'.
Security assessment
The diff is a resilience/availability guidance rewrite, not a fix for a specific vulnerability or incident. It does document data-protection practices with security relevance: verified backup restoration, RPO-aligned backup cadence, and monitoring for configuration drift such as disabled DynamoDB point-in-time recovery ('DynamoDB PITR is disabled (configuration drift)'), which supports data integrity and recoverability. No CVE, credential, authz, or exposure change is present, so severity is medium (hardening/backup guidance).
Evidence
+Document backup, restoration, and failover processes for your data, and test them regularly. A slow or failed data restore harms the advertising workload.
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp01.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp01.md index 4c1727e..2d11ac6 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp01.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel04-bp01.md @@ -7 +7 @@ -Implementation guidanceKey AWS servicesResources +Implementation guidanceResources @@ -9 +9 @@ Implementation guidanceKey AWS servicesResources -# ADVREL04-BP01 Through your CI/CD pipeline, employ end-to-end regression, performance, and canary testing +# ADVREL04-BP01 Create disaster recovery (DR) runbooks, and regularly test documented backup and restoration processes @@ -11 +11,38 @@ Implementation guidanceKey AWS servicesResources -Integrate comprehensive testing methodologies into CI/CD pipelines for advertising workloads. Monitor key metrics like 5xx errors and latency, especially in RTB systems, and respond quickly to issues through immediate engagement and fast rollbacks. +Document backup, restoration, and failover processes for your data, and test them regularly. A slow or failed data restore harms the advertising workload. + +**Desired outcome:** + + * DR runbooks for all Tier 1 advertising components are documented, tested, and kept current. + + * Backup restoration is validated quarterly, confirming that backups are intact and restoration completes within RTO. + + * Team members can execute DR procedures confidently under pressure. + + + + +**Common anti-patterns:** + + * Keeping DR procedures as undocumented knowledge instead of documented, step-by-step runbooks. + + * Configuring backups but never test-restoring them, so RTO and backup integrity are unverified. + + * Setting backup frequency that doesn't match the component's RPO (e.g., hourly backups for a 30-minute RPO). + + * Never exercising a full regional failover with partner coordination, so partner reconnection gaps surface only in a real disaster. + + + + +**Benefits of establishing this best practice:** + + * Tested runbooks let incident responders execute DR confidently and within RTO under pressure. + + * Quarterly restoration validation confirms backups are intact and recoverable. + + * Annual partner-coordinated failover exercises surface integration gaps before they matter. + + + + +**Level of risk exposed if this best practice is not established:** High @@ -15 +52,3 @@ Integrate comprehensive testing methodologies into CI/CD pipelines for advertisi -For RTB at scale, the primary reliability metrics for availability are 5xx internal errors and elevated latency. If these metrics are breached, do not wait for impacts to ad effectiveness. Instead, fail fast and revert changes until the root cause of the issue can be identified and addressed. +An advertising system runs inside a live traffic exchange that it doesn't control, so its disaster recovery differs from single-company DR. During an outage, SSP and DSP partners keep sending bid requests, timing out, and applying their own retry and circuit-breaking policies toward you. The order in which you restore components affects revenue directly. Stores and services on the bid path earn money the moment they return, and reporting and analytics can wait hours without costing anything. Sequence the runbook by revenue tier. + +Recovery puts the system under load. Partner traffic re-arrives at full rate against cold caches and freshly restored databases, which can push a half-recovered system back into failure. The runbook must describe how traffic returns (which partners, at what rate, and in what order) and how infrastructure returns. @@ -17 +56 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter -## Key AWS services +An exercise that restores databases and services without involving partners does not test how quickly partners deprioritize a bidder that stops answering, what their re-enablement process requires, or which partner contacts to notify for a gradual traffic ramp. Those relationships and procedures exist outside your AWS account. Partner contacts change and integration settings drift, so they decay unless exercises periodically include them. @@ -19 +58 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [AWS CodePipeline](https://aws.amazon.com/codepipeline/) is a fully-managed continuous delivery service +Perform a restore and measure it. Restore time, data integrity, and the gap between backup cadence and the tier's recovery point objective stay unknown until you run one. If a store's tolerable data loss is 30 minutes, backups must run at least that often. @@ -21 +60 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [AWS Fault Injection Service](https://aws.amazon.com/fis/) is a fully-managed service that simulates real-world failures +Runbooks drift out of date the same way. Every architecture change since the last validation leaves an untested step in the runbook. Size the validation cadence to how quickly the system changes. @@ -23 +62,9 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [AWS Step Functions](https://aws.amazon.com/step-functions/) +### Implementation steps + + 1. **Document step-by-step DR runbooks for each Tier 1 component:** Create runbooks that describe exact steps (with specific AWS CLI commands, console navigation paths, and validation checks) for: Aurora database failover to secondary region, DynamoDB restoration from point-in-time recovery, ElastiCache cluster replacement, and ECS/EKS service restoration from a blue environment. Each runbook should include estimated completion time and validation steps. + + 2. **Test backup restoration quarterly for all advertising data stores:** Schedule quarterly restoration exercises that actually restore from backup to a separate environment. Verify that: Aurora snapshots restore within the documented time, DynamoDB PITR restore creates a functional table with correct data, S3 versioned bucket data is recoverable, and ElastiCache can be rebuilt from the source database within acceptable time. + + 3. **Configure automated backup monitoring:** Create CloudWatch alarms that alert when: S3 lifecycle policies fail to run, RDS backup jobs fail, DynamoDB PITR is disabled (configuration drift), or backup storage costs are anomalous (indicating backup job failures). Route these alarms to the operations team as P1 alerts. + + 4. **Conduct annual full DR exercises with partner coordination:** Once a year, execute a full DR exercise that simulates a regional outage. Notify partner SSPs and DSPs in advance. Execute failover to the secondary region and verify that bid traffic successfully routes, latency is within acceptable bounds, and partner integrations reconnect. Measure total time from simulated failure declaration to full operational status. @@ -30 +77,16 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [Deployment strategies](https://docs.aws.amazon.com/whitepapers/latest/introduction-devops-aws/deployment-strategies.html) +**Related best practices:** + + * [ADVREL01-BP04 Implement chaos engineering practices](./advrel01-bp04.html) + + * [ADVREL05-BP05 Maintain independent operational readiness in each Region](./advrel05-bp05.html) + + + + +**Related documents:** + + * [Disaster recovery options in the cloud](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/disaster-recovery-options-in-the-cloud.html) + + * [Testing disaster recovery](https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/testing-disaster-recovery.html) + + @@ -32 +93,0 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [Canary deployments](https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/canary-deployments.html) @@ -34 +95 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [Use CloudWatch Synthetics to Monitor Sites, API Endpoints, Web Workflows, and More](https://aws.amazon.com/blogs/aws/new-use-cloudwatch-synthetics-to-monitor-sites-api-endpoints-web-workflows-and-more/) +**Related services:** @@ -36 +97 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [Performing canary deployments and metrics-driven rollback with Amazon managed Service for Prometheus and Flagger](https://aws.amazon.com/blogs/opensource/performing-canary-deployments-and-metrics-driven-rollback-with-amazon-managed-service-for-prometheus-and-flagger/index.html) + * [AWS Elastic Disaster Recovery](https://aws.amazon.com/disaster-recovery/) @@ -38 +99 @@ For RTB at scale, the primary reliability metrics for availability are 5xx inter - * [Testing and creating CI/CD pipelines for AWS Step Functions](https://aws.amazon.com/blogs/devops/testing-and-creating-ci-cd-pipelines-for-aws-step-functions-using-aws-codepipeline-and-aws-codebuild/index.html) + * [AWS Backup](https://aws.amazon.com/backup/) @@ -49 +110 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Change management +Failure, recovery, and capacity management @@ -51 +112 @@ Change management -ADVREL04-BP02 Deploy new code or resources in staggered phases, separated by sufficient time, to verify that the changes are successful +ADVREL04-BP02 Architect defensively against failures