AWS wellarchitected: Rewrite ADVREL03-BP03 on multi-AZ stateful dependency deployment
Summary
Restructures the reliability best practice from 'Configure databases to span across multiple Availability Zones' to 'Deploy stateful dependencies across multiple Availability Zones', adding desired outcomes, anti-patterns, tiered RTO/RPO guidance, cost trade-offs, seven implementation steps (Aurora Multi-AZ, DynamoDB PITR/global tables, ElastiCache Multi-AZ, MSK min.insync.replicas, removing single-AZ state), and updated related links.
Security assessment
The change is a reliability/availability documentation rewrite (RTO/RPO, failover, cross-AZ cost). It contains only tangentially security-adjacent items such as read-only secondary endpoints and point-in-time recovery, but no vulnerability, CVE, credential, IAM, or exposure fix is referenced, so it is not security documentation.
Evidence
+# ADVREL03-BP03 Deploy stateful dependencies across multiple Availability Zones
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp03.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp03.md index 9751b96..de8ea1c 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp03.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp03.md @@ -7 +7 @@ -Implementation guidanceKey AWS servicesResources +Implementation guidanceResources @@ -9 +9 @@ Implementation guidanceKey AWS servicesResources -# ADVREL03-BP03 Configure databases to span across multiple Availability Zones +# ADVREL03-BP03 Deploy stateful dependencies across multiple Availability Zones @@ -11 +11 @@ Implementation guidanceKey AWS servicesResources -Explore database configuration strategies for reliability and disaster recovery, such as periodic snapshots to warm standby solutions. Evaluate trade-offs between single-AZ and multi-AZ deployments, costs considerations, and specific recovery time objectives (RTO). +Deploy databases, caches, and message queues across multiple Availability Zones to match the RTO, RPO, and latency requirements of each advertising component tier. Evaluate trade-offs between single-AZ and multi-AZ deployments for each stateful dependency, with consideration for cost, failover speed, and the bid-response latency budget. @@ -13 +13 @@ Explore database configuration strategies for reliability and disaster recovery, -## Implementation guidance +**Desired outcome:** @@ -15 +15 @@ Explore database configuration strategies for reliability and disaster recovery, -Carefully consider the trade-offs between disaster recovery strategies when configuring databases in multi-AZ and single-AZ deployments. While multi-AZ deployments offer high availability, they can incur significant cross-AZ data transfer costs. + * Database configurations match the RTO and RPO requirements of the advertising components they serve. @@ -17 +17 @@ Carefully consider the trade-offs between disaster recovery strategies when conf -For cost-sensitive workloads, consider implementing a single-AZ database cluster with the following resilience strategies: + * Failover procedures are automated and tested, not discovered for the first time during a production incident. @@ -19 +19 @@ For cost-sensitive workloads, consider implementing a single-AZ database cluster - 1. **Periodic snapshots:** Implement frequent automated snapshots of your database. This approach provides point-in-time recovery capabilities with a relatively low RTO, typically in the range of 15-60 minutes, depending on the database size and recovery process. + * The recovery strategy selected (from periodic snapshots to multi-AZ hot standby) is proportional to the business impact of data loss. @@ -21 +21 @@ For cost-sensitive workloads, consider implementing a single-AZ database cluster - 2. **Read replicas:** Deploy read replicas in a different Availability Zone. While this incurs some cross-AZ data transfer costs, it's generally less expensive than a full multi-AZ deployment. In case of a primary Availability Zone failure, promote the read replica to become the new primary. This can reduce RTO to between five and 15 minutes. + * Failover for all Tier 1 stateful dependencies completes within the bid response latency budget. @@ -23 +22,0 @@ For cost-sensitive workloads, consider implementing a single-AZ database cluster - 3. **Cold standby:** Maintain a stopped database instance in another Availability Zone, and periodically update it with snapshots. This approach balances cost and recovery time, with an RTO of approximately 10-30 minutes. @@ -26,0 +26 @@ For cost-sensitive workloads, consider implementing a single-AZ database cluster +**Common anti-patterns:** @@ -28 +28 @@ For cost-sensitive workloads, consider implementing a single-AZ database cluster -For mission-critical applications, where minimal downtime is essential, consider: + * Applying one database HA pattern uniformly instead of matching it to each tier's RTO/RPO and business impact. @@ -30 +30 @@ For mission-critical applications, where minimal downtime is essential, consider - 1. **Warm standby:** Keep an active, scaled-down secondary database in another Availability Zone continuously updated using asynchronous replication. This approach offers a lower RTO (between one and five minutes), but at a higher cost than cold standby. + * Enabling multi-AZ everywhere without accounting for cross-AZ data transfer cost on high-volume tables. @@ -31,0 +32 @@ For mission-critical applications, where minimal downtime is essential, consider + * Connecting applications to instance endpoints rather than the cluster endpoint, defeating automatic failover. @@ -32,0 +34 @@ For mission-critical applications, where minimal downtime is essential, consider + * Assuming failover works without testing it, discovering the actual failover duration for the first time during a production incident. @@ -33,0 +36 @@ For mission-critical applications, where minimal downtime is essential, consider + * Storing critical bid state only in local instance storage or single-AZ volumes while assuming compute redundancy is sufficient. @@ -35 +38 @@ For mission-critical applications, where minimal downtime is essential, consider -Choose the strategy that best aligns with your specific RTO requirements and budget constraints. Implement and regularly test your chosen disaster recovery process to verify that it meets your RTO targets. + * Running caching layers (ElastiCache, Memcached) in single-AZ mode, so a zone failure removes both database failover protection and the cache layer simultaneously. @@ -37 +39,0 @@ Choose the strategy that best aligns with your specific RTO requirements and bud -For AdTech customers who require multi-region deployment for global resilience, use services like Amazon Aurora Global Database or Amazon DynamoDB global tables. These services provide Region-wide resilience with minimal impact on performance and manageable costs. @@ -39 +40,0 @@ For AdTech customers who require multi-region deployment for global resilience, -Regularly review and optimize your database architecture as your workload and requirements evolve. Always weigh the costs of potential downtime against the ongoing expenses of more resilient configurations. @@ -41 +41,0 @@ Regularly review and optimize your database architecture as your workload and re -## Key AWS services @@ -43 +43 @@ Regularly review and optimize your database architecture as your workload and re - * [Amazon Relational Database Service (Amazon RDS)](https://aws.amazon.com/rds/) provides a Multi-AZ deployment option +**Benefits of establishing this best practice:** @@ -45 +45 @@ Regularly review and optimize your database architecture as your workload and re - * [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) + * Database configurations match the RTO and RPO requirements of the components they serve, avoiding over- and under-provisioning. @@ -47 +47,26 @@ Regularly review and optimize your database architecture as your workload and re - * [Amazon Aurora](https://aws.amazon.com/rds/aurora/) + * Automated, tested failover keeps the bid path available during Availability Zone disruptions. + + * Recovery strategy cost is proportional to the business impact of data loss for each tier. + + + + +**Level of risk exposed if this best practice is not established:** High + +## Implementation guidance + +Treating advertising data stores as one single class can produce availability or over-engineering issues. Tiers exist because data staleness has a different business meaning in each store. A frequency-cap or user-profile store serves the live auction. If it lags or becomes unavailable, the system overserves ads within seconds. Overserving spends already-exhausted budgets and breaches contractual caps. Both the acceptable data loss and the acceptable downtime are therefore measured in seconds to a minute. + +People can directly edit campaign management data. A trafficker who can't change a line item for ten minutes loses productive time without creating compliance exposure, so minutes of recovery are tolerable. Analytics stores serve no live traffic at all. A warehouse restored hours later reproduces the same reports, so paying for instant recovery there buys nothing. + +Synchronous multi-AZ replication and globally replicated tables add an ongoing premium. Pay it where a minute of unavailability costs real auction revenue or creates overspend liability, and skip it on data that isn't read by live traffic. + +Each service bills that premium differently. A Multi-AZ relational deployment roughly doubles instance and storage cost because you pay for the standby. The replication traffic between primary and standby isn't separately billed. A globally replicated table instead multiplies write cost by the number of replica Regions. Each write is billed as a replicated write unit in every Region. No separate cross-Region transfer fee applies. Either way, the cost is ongoing and scales with write volume. + +Under-protecting Tier 1 becomes visible quickly. Paying Tier 1 prices for Tier 3 data is harder to spot and happens more often, because a default template applied the same deployment everywhere. Multi-Region replication is the next step above multi-AZ in the same trade-off. It protects globally served data against Regional impairment at a higher premium. Only globally read stores (campaign configuration and audience definitions) justify that premium. + +Replication and failover protect against infrastructure loss. Bad data is a separate problem. A corrupt write, an errant migration, or an application bug replicates to the standby as faithfully as a good write. Failing over then reaches the same damaged data. Point-in-time recovery and snapshots solve that separate problem. Even at the highest tier, pair replication with an independent recovery path. + +Choose each store's tier jointly with the revenue owners, not the database team alone. Tier placement depends on the business impact of data loss and cannot be automated. + +### Implementation steps @@ -49 +74,13 @@ Regularly review and optimize your database architecture as your workload and re - * [Amazon ElastiCache](https://aws.amazon.com/elasticache/) + 1. **Configure Aurora Multi-AZ for Tier 1 auction databases:** Deploy Aurora clusters with one writer and one reader replica in a different Availability Zone. Enable automatic failover. Test failover by using AWS FIS to simulate an Availability Zone failure and measuring actual failover duration. Configure application connection strings to use the cluster endpoint (not instance endpoint) so that failover happens automatically without application changes. + + 2. **Configure DynamoDB for high-availability bid path data stores:** For user profile and frequency cap data, use DynamoDB with on-demand capacity and point-in-time recovery enabled. For global campaigns, enable DynamoDB global tables. Set up DynamoDB TTL on bid event records to manage table size and query performance. Configure CloudWatch alarms on DynamoDB throttled requests to detect capacity issues before they affect bid processing. + + 3. **Implement single-AZ database resilience for cost-sensitive Tier 2 components:** For campaign management databases where RPO/RTO requirements permit, implement periodic snapshots (every 15 minutes) plus a read replica in a different Availability Zone. Use the read replica for promotion to primary in case of Availability Zone failure (5–15 minute RTO). Document the promotion procedure and include it in the DR runbook. + + 4. **Use Aurora Global Database for lowest-latency Multi-Region reads:** For advertising data that must be read from multiple regions (campaign configurations, audience segment definitions, brand safety rules), deploy Aurora Global Database with a primary region and 1–2 secondary regions. Measure actual read latency from each Region and verify it meets your bid-path requirements. Configure the secondary regions as read-only endpoints to block accidental writes. + + 5. **Make message queues and caching layers Multi-AZ:** Amazon SQS is inherently Multi-AZ. If using Amazon MSK or self-managed Kafka, confirm brokers span at least 2 Availability Zones with min.insync.replicas=2. Enable ElastiCache for Redis Multi-AZ with automatic failover. Size each Availability Zone's cache to handle the full workload (not just its proportional share) so the surviving Availability Zone can absorb traffic immediately after failover. + + 6. **Remove single-AZ state from the bid path:** Audit the bid pipeline for any state held only in local instance storage, single-AZ EBS volumes, or in-memory process state without external persistence. Migrate critical state to Multi-AZ stores (DynamoDB, ElastiCache Multi-AZ, or replicated EBS with Multi-Attach). + + 7. **Test failover against the bid response latency budget:** Test that database and cache failover completes within your bid response latency budget. RDS Multi-AZ DB instance failover typically takes 60 to 120 seconds (crash recovery on cold standby). Multi-AZ DB cluster deployments (MySQL/PostgreSQL) fail over in under 35 seconds. ElastiCache Multi-AZ failover completes in under 30 seconds. Verify these timings against your bid deadline requirements using AWS FIS experiments. @@ -55,0 +93,11 @@ Regularly review and optimize your database architecture as your workload and re +**Related best practices:** + + * [ADVREL04-BP04 Design your workloads to withstand failures of individual components](./advrel04-bp04.html) + + * [ADVREL03-BP01 Use a full regional deployment for compute resources through Auto Scaling groups and compute container orchestrators](./advrel03-bp01.html) + + + + +**Related documents:** + @@ -58 +106,2 @@ Regularly review and optimize your database architecture as your workload and re - * [Protect critical workload with Pod Disruption Budgets](https://docs.aws.amazon.com/eks/latest/best-practices/application.html#_recommendations_3) + * [Using Amazon Aurora Global Database to build resilient multi-Region applications](https://aws.amazon.com/blogs/database/use-amazon-aurora-global-database-to-build-resilient-multi-region-applications/) + @@ -60 +108,0 @@ Regularly review and optimize your database architecture as your workload and re - * [Using Amazon Aurora Global Database](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-global-database.html) @@ -62 +109,0 @@ Regularly review and optimize your database architecture as your workload and re - * [Amazon DynamoDB global tables](https://aws.amazon.com/dynamodb/global-tables/) @@ -64 +111,5 @@ Regularly review and optimize your database architecture as your workload and re - * [What is Amazon Relational Database Service (Amazon RDS)?](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Welcome.html) +**Related services:** + + * [Amazon Aurora](https://aws.amazon.com/rds/aurora/) + + * [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) @@ -66 +117 @@ Regularly review and optimize your database architecture as your workload and re - * [Multi-AZ DB instance deployments for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) + * [Amazon RDS](https://aws.amazon.com/rds/) @@ -79 +130 @@ ADVREL03-BP02 Choose AWS Regions that meet your legal and disaster recovery requ -ADVREL03-BP04 Reserve appropriate capacity of services in the supported Regions +ADVREL03-BP04 Reserve appropriate capacity of services in the supported regions