AWS Security ChangesHomeSearch

AWS wellarchitected: ADVREL02-BP01: fast-fail, rate limiting, and WAF guidance for bid path

Service: wellarchitected · 2026-11-11 · Documentation medium

File: wellarchitected/latest/video-streaming-advertising-lens/advrel02-bp01.md · Type: network

Summary

Rewrites the ADVREL02-BP01 best practice on handling latency-sensitive service failures. Adds desired outcomes, anti-patterns, risk level, and five implementation steps covering bid-path vs post-bid service classification, API Gateway token-bucket throttling with WAF rate-based rules, per-service timeout budgets, SQS/DLQ for asynchronous post-bid processing, and CloudWatch alarms on queue depth and throttle counts. Also fixes wording (backoff, latency-sensitive) and updates related links.

Security assessment

The added implementation step explicitly prescribes AWS WAF rate-based rules for IP-level protection and per-client API Gateway throttling to guard against a single misbehaving client, plus SQS dead-letter queues and CloudWatch alarms on throttle/timeout rates. This is hardening guidance against resource exhaustion and abusive traffic rather than a fix for a specific vulnerability or incident, so it is security-adjacent documentation (medium) and not tied to a CVE.

Evidence

  2. **Implement token bucket rate limiting on bid-path API endpoints:** Configure Amazon API Gateway throttling with per-client rate limits based on traffic volume agreements. Set burst limits conservatively to guard against a single misbehaving client affecting others. Use WAF rate-based rules for IP-level protection. Configure limits to allow 120% of contracted QPS to accommodate burst traffic while blocking clearly anomalous spikes.

Diff

diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel02-bp01.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel02-bp01.md
index c26a5d2..43cde2e 100644
--- a//wellarchitected/latest/video-streaming-advertising-lens/advrel02-bp01.md
+++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel02-bp01.md
@@ -7 +7 @@
-Implementation guidanceKey AWS servicesResources
+Implementation guidanceResources
@@ -9 +9 @@ Implementation guidanceKey AWS servicesResources
-# ADVREL02-BP01 To allow fast and graceful failure of latency-sensitive services, avoid exponential backing off and retry
+# ADVREL02-BP01 To allow fast and graceful failure of latency-sensitive services, avoid exponential backoff and retry
@@ -11 +11,36 @@ Implementation guidanceKey AWS servicesResources
-With real-time bidding systems, your workload must handle failures in latency-sensitive services. Traditional exponential backoff and retry mechanisms should be avoided. Instead, opt for fast-fail approaches and appropriate rate-limiting techniques to maintain service responsiveness. 
+With real-time bidding, your workload must handle failures in latency-sensitive services within the 100 ms bid response window. Avoid exponential backoff and retry. Use fast-fail approaches and rate limiting to keep services responsive. 
+
+**Desired outcome:**
+
+  * Latency-sensitive service failures are detected and handled within the bid response time budget rather than accumulating retry queues. 
+
+  * Rate limiting reduces the risk of resource exhaustion while fairly distributing resources among clients. 
+
+  * The auction completes and returns the best available bids even when individual services are degraded. 
+
+
+
+
+**Common anti-patterns:**
+
+  * Using exponential backoff with 5-second maximum delays on bid path services. 
+
+  * Not distinguishing between bid path services (must fail fast) and post-bid analytics services (can retry aggressively asynchronously). 
+
+  * Setting rate limits too low, so legitimate requests get rejected or delayed and ad serving effectiveness drops. 
+
+
+
+
+**Benefits of establishing this best practice:**
+
+  * Bid response latency stays within SLO even when downstream services are degraded. 
+
+  * Rate limiting helps protect services from being overloaded while maintaining fair resource distribution. 
+
+  * Clear failure handling behavior is predictable and testable. 
+
+
+
+
+**Level of risk exposed if this best practice is not established:** High 
@@ -15 +50,7 @@ With real-time bidding systems, your workload must handle failures in latency-se
-Operating within 100 ms real-time bidding contracts, a single throttle and retry of five seconds can result in many failed bids and potentially insurmountable retry queues. Avoid this by adapting retries to fail fast. Regulate request rates using algorithms, such as token buckets, leaky buckets, or fixed window counters, or use managed service features, like Amazon API Gateway's request throttling. Rate limiting helps prevent resource exhaustion and fairly distributes resources among clients or services. Know the trade-offs: while rate limiting can be an effective way to protect a service from being overloaded, it can also potentially make the service less reliable if not implemented carefully. For example, if the rate limits are set too low, legitimate requests may be rejected or delayed, leading to reduced availability or responsiveness of the service. 
+In a 100 ms bid response window, a single throttle and 5-second retry can produce many failed bids and retry queues that never drain. Regulate request rates using token buckets, leaky buckets, or fixed window counters, or use a managed service such as Amazon API Gateway request throttling. 
+
+Rate limiting reduces resource exhaustion and distributes resources fairly among clients or services. Limits set too low reject or delay legitimate requests and reduce availability or responsiveness. 
+
+For bid path services, attempt the call once. If it fails within the latency budget, proceed with cached data, default values, or omit the bid. For post-bid services such as analytics, billing, and frequency updates, use aggressive retry with Amazon SQS and a dead-letter queue (DLQ). These services run asynchronously outside the bid response window. 
+
+Amazon API Gateway implements the token bucket algorithm to throttle requests. Amazon SQS and Amazon Kinesis can buffer requests to smooth request rates. AWS WAF can apply rate limiting and throttle specific API consumers. 
@@ -17 +58 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-## Key AWS services
+### Implementation steps
@@ -19 +60 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-  * [Amazon API Gateway](https://aws.amazon.com/api-gateway/) implements the token bucket algorithm to throttle requests according to account and region limits 
+  1. **Distinguish bid-path services from post-bid services:** Create a service inventory that classifies each advertising service as bid-path (must fail fast within the response window) or post-bid (can retry aggressively asynchronously). Apply different error handling strategies to each class: fast-fail with graceful degradation for bid-path, reliable retry with DLQ for post-bid. 
@@ -21 +62 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-  * [Amazon Simple Queue Service (Amazon SQS)](https://aws.amazon.com/sqs/) and Amazon Kinesis can buffer requests to smooth out the request rate 
+  2. **Implement token bucket rate limiting on bid-path API endpoints:** Configure Amazon API Gateway throttling with per-client rate limits based on traffic volume agreements. Set burst limits conservatively to guard against a single misbehaving client affecting others. Use WAF rate-based rules for IP-level protection. Configure limits to allow 120% of contracted QPS to accommodate burst traffic while blocking clearly anomalous spikes. 
@@ -23 +64,5 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-  * [AWS WAF](https://aws.amazon.com/waf/) can also be used to implement rate limiting and throttle specific API consumers 
+  3. **Add per-service timeout budgets on the bid path:** For each bid-path service call, set a timeout equal to the service's allocated portion of the total bid response budget (typically 5–15 ms for enrichment lookups, 10–20 ms for targeting data retrieval). If the service doesn't respond within its budget, proceed with a cache hit, default value, or empty response. Log all timeout events with service name and elapsed time for capacity planning. 
+
+  4. **Use SQS for reliable post-bid asynchronous processing:** Route all post-bid events (impression beacons, click tracking, frequency cap updates, analytics) through SQS queues with maxReceiveCount and DLQ configured. Use standard queues for high-throughput events that tolerate at-least-once delivery, FIFO queues for ordered events (such as bid events that must be processed in auction sequence). Configure visibility timeout equal to your expected processing time × 2 to allow retries. 
+
+  5. **Monitor queue depths and timeout rates as leading indicators:** Configure CloudWatch alarms on SQS queue depth (growing queue indicates processing backlog), API Gateway throttle count (increasing throttles indicate capacity constraints), and bid-path service timeout rate (increasing timeouts indicate service degradation). Use these metrics to proactively scale services before they become auction-affecting bottlenecks. 
@@ -30 +75,6 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-**Related documentation:**
+**Related best practices:**
+
+  * [ADVREL04-BP02 Architect defensively against failures](./advrel04-bp02.html)
+
+  * [ADVREL02-BP03 Avoid scale mismatch of both internal services and external partners](./advrel02-bp03.html)
+
@@ -32 +82,5 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
-  * [Implementing layers of admission control](https://aws.amazon.com/builders-library/fairness-in-multi-tenant-systems/)
+
+
+**Related documents:**
+
+  * [Implementing layers of admission control](https://aws.amazon.com/builders-library/implementing-health-checks/)
@@ -38,0 +93,11 @@ Operating within 100 ms real-time bidding contracts, a single throttle and retry
+**Related services:**
+
+  * [Amazon API Gateway](https://aws.amazon.com/api-gateway/)
+
+  * [Amazon SQS](https://aws.amazon.com/sqs/)
+
+  * [AWS WAF](https://aws.amazon.com/waf/)
+
+
+
+
@@ -45 +110 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please
-Latency sensitive advertising
+Latency-sensitive advertising