AWS wellarchitected: ADVREL03-BP01: expand multi-AZ resilience guidance for ad bidding
Summary
Rewrites the ADVREL03-BP01 best practice for full regional/multi-AZ compute deployment: adds desired outcomes, common anti-patterns, benefits, a 'High' risk level, and five implementation steps (Auto Scaling groups spanning 3 AZs with AvailabilityZoneRebalancing, EKS pod anti-affinity, zone-aware NLB load balancing, Karpenter zone-balanced provisioning, FIS AZ-failure experiments), plus related best practices/documents/services links and minor title/link cleanup.
Security assessment
The change is reliability/availability guidance (AZ distribution, load balancing, capacity, failure-injection testing) for real-time bidding workloads; it contains no authentication, encryption, credential, IAM, or vulnerability-fix content, so it is not security documentation and addresses no specific weakness or incident.
Evidence
+**Level of risk exposed if this best practice is not established:** High
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp01.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp01.md index c192927..ead7366 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp01.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel03-bp01.md @@ -7 +7 @@ -Implementation guidanceKey AWS servicesResources +Implementation guidanceResources @@ -9 +9 @@ Implementation guidanceKey AWS servicesResources -# ADVREL03-BP01 Use a full Regional deployment for compute resources through Auto Scaling groups and compute container orchestrators +# ADVREL03-BP01 Use a full regional deployment for compute resources through Auto Scaling groups and compute container orchestrators @@ -11 +11,34 @@ Implementation guidanceKey AWS servicesResources -Deploy compute resources across multiple Availability Zones (AZs) and Regions to enhance application resilience. Implement zone-aware architectures to optimize performance and manage costs, and focus on intra-AZ communication and load balancing configurations. +Deploy compute resources across multiple Availability Zones and Regions to improve application resilience. Implement zone-aware architectures to optimize performance and manage costs, and focus on intra-AZ communication and load balancing configurations. + +**Desired outcome:** + + * Compute resources are distributed across multiple Availability Zones behind zone-aware load balancers. + + * Availability Zone-level failures trigger automatic traffic redistribution without manual intervention. + + * Zone-aware load balancing minimizes cross-AZ data transfer costs for intra-cluster communication. + + + + +**Common anti-patterns:** + + * Distributing compute across Availability Zones but using default cross-zone load balancing settings, which routes all traffic evenly across Availability Zones and adds unnecessary cross-AZ transfer costs for stateful advertising services. + + * Not implementing pod anti-affinity rules in EKS, so all replicas of a critical bidding service can end up in a single Availability Zone. + + + + +**Benefits of establishing this best practice:** + + * Availability Zone-level failures are absorbed automatically without dropped bids. + + * Cross-AZ data transfer costs are minimized through zone-aware communication. + + * Capacity scales horizontally across Availability Zones to absorb uneven load distribution. + + + + +**Level of risk exposed if this best practice is not established:** High @@ -15 +48,19 @@ Deploy compute resources across multiple Availability Zones (AZs) and Regions to -Increase resiliency of real-time advertising applications by distributing resources across multiple Availability Zones or Regions, but maintain awareness of cross-AZ and cross-Region data transfer costs. When you use a full Regional deployment, implement zone-aware architectures within each Region to optimize performance and costs. When distributing resources across multiple Availability Zones for resilience, implement logic to prefer intra-AZ communication, when possible, and use features like AZ-aware load balancing to minimize cross-AZ traffic. By being zone-aware, companies can reduce costs and improve performance even when they need to operate in multiple Regions. +An exchange gives a bidder roughly 100 milliseconds end to end. Network transit to and from the exchange consumes part of that, so the decisioning logic often has single-digit milliseconds to spend. Multi-AZ design for that bid path answers a different question than it does for a typical web application. + +Every cross-AZ hop inside the path adds roughly a millisecond one way at best. AWS Availability Zone latency guidance puts the median at about 1.5 milliseconds one way. A path that crosses zone boundaries several times can spend single-digit milliseconds on network transit alone. That amount is negligible for a management console and meaningful for an auction. + +AWS bills cross-AZ data transfer per gigabyte in each direction. Without Availability Zone affinity, a bidder handling billions of requests per day crosses Availability Zone boundaries billions of times. A design decision that is invisible at normal traffic levels becomes one of the larger infrastructure costs at RTB scale. + +Place complete, independently functional bidding capacity in every zone. The loss of a zone then removes only a fraction of the fleet. Each request enters a zone, is answered in that zone, and leaves. Availability Zone affinity in load balancing enforces that path. Spreading an individual request across zones buys no additional resilience, because whole replicas already exist elsewhere. It also pays the latency and transfer cost on every auction. + +Services behind an ALB, such as campaign management consoles, reporting APIs, or creative upload endpoints, have no auction clock and orders of magnitude less volume. The affinity rule is scoped to deadline-bound, high-volume traffic, so it inverts for these services. Uneven load is the primary risk for them. + +Turning cross-zone off makes each load balancer node serve only the targets in its own zone. Unequal target counts after a scale-in event then translate into unequal per-target load. At low request volume, the statistical variance across a small number of requests concentrates that load further. Turning it off also invalidates any target stickiness on the group, which ALB doesn't support when cross-zone load balancing is off. For those workloads, even distribution across all healthy targets matters more than keeping traffic zone-local, so cross-zone load balancing stays on. + +Carry that distinction into the steps. An auction deadline and per-request economics call for keeping traffic in-zone, and utilization smoothness on modest traffic calls for spreading it. + +### Implementation steps + + 1. **Configure Auto Scaling groups to span at least 3 Availability Zones:** Create Auto Scaling groups with subnets in all Availability Zones within your deployment region. Set AvailabilityZoneRebalancing to Enabled to redistribute instances after Availability Zone recovery. Configure minimum healthy percentage (85%) so that the fleet remains operational during rolling updates or Availability Zone failures. + + 2. **Implement EKS pod anti-affinity for critical bidding services:** Add podAntiAffinity rules with topologyKey: topology.kubernetes.io/zone to all critical advertising service deployments. Use preferredDuringSchedulingIgnoredDuringExecution for standard services and requiredDuringSchedulingIgnoredDuringExecution for the most critical Tier 1 auction services to guarantee Availability Zone distribution. @@ -17 +68 @@ Increase resiliency of real-time advertising applications by distributing resour -## Key AWS services + 3. **Configure zone-aware load balancing:** For NLB serving RTB bid traffic, disable cross-zone load balancing to keep traffic within each Availability Zone and avoid cross-AZ data transfer charges. Configure NLB with one IP per Availability Zone and use Route 53 latency-based routing or Availability Zone DNS to route clients to the NLB endpoint in their Availability Zone. Verify Availability Zone traffic distribution using CloudWatch metrics per target group. @@ -19 +70 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Amazon EC2 Auto Scaling](https://aws.amazon.com/autoscaling/) groups can be configured to span multiple AZs + 4. **Deploy Karpenter with zone-balanced node provisioning:** Configure Karpenter NodePool with requirements specifying all Availability Zones in the Region. Set consolidationPolicy: WhenUnderutilized for batch workloads and consolidationPolicy: WhenEmpty for latency-sensitive bidding workloads to avoid consolidation during active campaigns. @@ -21 +72 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/) clusters can also be deployed across multiple AZs + 5. **Validate Availability Zone failure recovery through FIS experiments:** Create an FIS experiment that terminates all EC2 instances in one Availability Zone simultaneously. Verify that Auto Scaling replaces instances in the remaining Availability Zones within the configured RTO. Monitor bid response latency and error rate during the experiment to confirm that the Availability Zone failure doesn't cause SLO violations. @@ -28 +79,15 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Regions and Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) +**Related best practices:** + + * [ADVREL01-BP03 Architect for variable demand](./advrel01-bp03.html) + + * [ADVREL03-BP04 Reserve appropriate capacity of services in the supported regions](./advrel03-bp04.html) + + + + +**Related documents:** + + * [Distribute instances across Availability Zones](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html) + + * [Static stability using Availability Zones](https://aws.amazon.com/builders-library/static-stability-using-availability-zones/) + @@ -30 +94,0 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Distribute instances across Availability Zones](https://docs.aws.amazon.com/autoscaling/ec2/userguide/auto-scaling-benefits.html#arch-AutoScalingMultiAZ) @@ -32 +95,0 @@ Increase resiliency of real-time advertising applications by distributing resour - * [EC2 Instance Meta-Data Retrieval](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) @@ -34 +97 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Creating Kubernetes Auto Scaling Groups for Multiple Availability Zones | Containers ](https://aws.amazon.com/blogs/containers/amazon-eks-cluster-multi-zone-auto-scaling-groups/index.html) +**Related services:** @@ -36 +99 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Add an Availability Zone - Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-add-az-console.html) + * [Amazon EC2 Auto Scaling](https://aws.amazon.com/ec2/autoscaling/) @@ -38 +101 @@ Increase resiliency of real-time advertising applications by distributing resour - * [Simplify node lifecycle with managed node groups - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html) + * [Amazon EKS](https://aws.amazon.com/eks/) @@ -49 +112 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Design for single- and Multi-Region deployments +Single and Multi-Region deployments