AWS Security ChangesHomeSearch

AWS wellarchitected: Rewrote ADVREL08-BP02 to focus on data collection pipelines

Service: wellarchitected · 2026-11-11 · Documentation low

File: wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md

Summary

Updated the best practice to emphasize reliable data collection, validation, dead-letter queues, and reconciliation for measurement accuracy, replacing previous content on data consistency and collaboration workflows.

Security assessment

The change is a documentation rewrite focused on reliability and data integrity for advertising measurement, not on security vulnerabilities or features.

Evidence

+# ADVREL08-BP02 Establish reliable data collection and validation pipelines for measurement accuracy

Diff

diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md
index 6cabc59..06d1f61 100644
--- a//wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md
+++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md
@@ -7 +7 @@
-Implementation guidanceKey AWS servicesResources
+Implementation guidanceResources
@@ -9 +9 @@ Implementation guidanceKey AWS servicesResources
-# ADVREL08-BP02 Maintain data consistency and availability across collaboration workflows
+# ADVREL08-BP02 Establish reliable data collection and validation pipelines for measurement accuracy
@@ -11 +11 @@ Implementation guidanceKey AWS servicesResources
-Data consistency and availability is critical when working with multiple stakeholders or workflows. Implement tools like versioning, logging, and health checks to verify that data remains consistent and available. 
+Build reliable data collection and validation pipelines with real-time monitoring, automated reconciliation, and recovery procedures to maintain measurement accuracy in advertising systems. 
@@ -13 +13,27 @@ Data consistency and availability is critical when working with multiple stakeho
-## Implementation guidance
+**Desired outcome:**
+
+  * Measurement events are collected with at-least-once delivery guarantees and deduplicated before processing. 
+
+  * Data quality checks run at each pipeline stage to catch anomalies before they propagate to reporting. 
+
+  * Failed measurement events are processed from dead-letter queues rather than discarded. 
+
+
+
+
+**Common anti-patterns:**
+
+  * Collecting measurement events without at-least-once delivery guarantees, so events are lost on failure. 
+
+  * Discarding failed events instead of routing them to a dead-letter queue for reprocessing. 
+
+  * Running data quality checks only at the end, so anomalies propagate to reporting before detection. 
+
+  * No reconciliation against third-party provider totals, so collection drift accumulates undetected. 
+
+
+
+
+**Benefits of establishing this best practice:**
+
+  * At-least-once collection with deduplication supports measurement completeness without double counting. 
@@ -15 +41 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Implement versioning for collaborative datasets and schemas. 
+  * Stage-by-stage data quality checks catch anomalies before they reach reporting. 
@@ -17 +43 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Use transaction logs for tracking privacy computation state. 
+  * Dead-letter queue processing recovers failed events rather than losing them. 
@@ -19 +44,0 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Configure cross-Region replication for critical data stores. 
@@ -21 +45,0 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Implement idempotency for matching operations. 
@@ -23 +46,0 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Set up health checks for collaboration service endpoints. 
@@ -25 +48 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Use read replicas for high-availability data access. 
+**Level of risk exposed if this best practice is not established:** High 
@@ -27 +50 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Configure automated rollback procedures for failed operations. 
+## Implementation guidance
@@ -28,0 +52 @@ Data consistency and availability is critical when working with multiple stakeho
+The longer an error remains, the higher its corresponding cost for correction. Once an event enters the Kinesis pipeline, it spreads to aggregation, reporting, and billing. Fixing it there requires a full AWS Glue ETL reprocessing run and possible advertiser billing adjustments. Validate close to collection to help avoid such steadily rising costs. 
@@ -29,0 +54 @@ Data consistency and availability is critical when working with multiple stakeho
+At-least-once delivery accepts duplicates in exchange for never losing an event. Pair it with DynamoDB-backed deduplication to remove the duplicates the guarantee allows. Together, they give completeness without double counting. 
@@ -30,0 +56 @@ Data consistency and availability is critical when working with multiple stakeho
+Accept events to S3 and route validation failures to an SQS dead-letter queue. Store the original events in an S3 bucket to preserve valid data when a client is misconfigured, and use a dead-letter queue to block invalid events from moving downstream undetected. 
@@ -32 +58 @@ Data consistency and availability is critical when working with multiple stakeho
-## Key AWS services
+A KCL consumer can fail without raising an alarm, and small persistent losses cause measurement drift over time. Reconciliation against third-party totals eventually exposes the drift, but by then some data might be unrecoverable. Assign an owner to the reconciliation cadence who acts on its findings. Add stage-by-stage data quality checks so that owner detects anomalies earlier than a final check would. 
@@ -34 +60 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Amazon DynamoDB 
+### Implementation steps
@@ -36 +62 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Amazon S3 
+  1. **Deploy Kinesis Data Streams for measurement event collection:** Create Kinesis shards sized for your peak measurement event throughput (each shard handles 1,000 records/second). Configure Kinesis Consumer Library (KCL) with automatic checkpoint management. Set ShardIteratorType to TRIM_HORIZON during recovery scenarios so that no events are missed. 
@@ -38 +64 @@ Data consistency and availability is critical when working with multiple stakeho
-  * AWS Lambda 
+  2. **Implement dead-letter queue processing for all measurement consumers:** Configure DLQ on all SQS queues and Kinesis stream consumers. Set DLQ depth alarms at 100 messages (immediate investigation required). Deploy Lambda functions that process DLQ messages on a 15-minute schedule, attempting reprocessing with exponential backoff. Generate reports on DLQ processing for data loss tracking. 
@@ -40 +66 @@ Data consistency and availability is critical when working with multiple stakeho
-  * Amazon CloudWatch 
+  3. **Run automated reconciliation between measurement sources:** Deploy a daily AWS Glue job that compares measurement totals from your collection pipeline against third-party verification provider reports. Flag discrepancies exceeding 2% for investigation. Store reconciliation results in S3 and publish summary metrics to CloudWatch for trend analysis. 
@@ -47 +73,27 @@ Data consistency and availability is critical when working with multiple stakeho
-  * [Guidance for Maximum Data Availability Architecture on AWS](https://aws.amazon.com/solutions/guidance/maximum-data-availability-architecture-on-aws/)
+**Related best practices:**
+
+  * [ADVREL08-BP01 Implement redundant ad verification systems with automated failover mechanisms](./advrel08-bp01.html)
+
+  * [ADVREL02-BP02 Implement a caching strategy](./advrel02-bp02.html)
+
+
+
+
+**Related documents:**
+
+  * [Avoiding insurmountable queue backlogs - Amazon Builders' Library](https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/)
+
+  * [Making retries safe with idempotent APIs - Amazon Builders' Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-apis/)
+
+
+
+
+**Related services:**
+
+  * [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/)
+
+  * [Amazon SQS](https://aws.amazon.com/sqs/)
+
+  * [Amazon S3](https://aws.amazon.com/s3/)
+
+  * [AWS Glue](https://aws.amazon.com/glue/)
@@ -49 +101 @@ Data consistency and availability is critical when working with multiple stakeho
-  * [CAP theorem](https://docs.aws.amazon.com/whitepapers/latest/availability-and-beyond-improving-resilience/cap-theorem.html)
+  * [Amazon EMR](https://aws.amazon.com/emr/)
@@ -60 +112 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please
-ADVREL08-BP01 Design resilient architectures with privacy-preserving fault tolerance
+ADVREL08-BP01 Implement redundant ad verification systems with automated failover mechanisms
@@ -62 +114 @@ ADVREL08-BP01 Design resilient architectures with privacy-preserving fault toler
-ADVREL08-BP03 Implement secure and privacy-preserving recovery mechanisms for collaboration workloads
+Prebid Server resilience