AWS wellarchitected: Rewrote ADVREL08-BP02 to focus on data collection pipelines
Summary
Updated the best practice to emphasize reliable data collection, validation, dead-letter queues, and reconciliation for measurement accuracy, replacing previous content on data consistency and collaboration workflows.
Security assessment
The change is a documentation rewrite focused on reliability and data integrity for advertising measurement, not on security vulnerabilities or features.
Evidence
+# ADVREL08-BP02 Establish reliable data collection and validation pipelines for measurement accuracy
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md b/wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md index 6cabc59..06d1f61 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advrel08-bp02.md @@ -7 +7 @@ -Implementation guidanceKey AWS servicesResources +Implementation guidanceResources @@ -9 +9 @@ Implementation guidanceKey AWS servicesResources -# ADVREL08-BP02 Maintain data consistency and availability across collaboration workflows +# ADVREL08-BP02 Establish reliable data collection and validation pipelines for measurement accuracy @@ -11 +11 @@ Implementation guidanceKey AWS servicesResources -Data consistency and availability is critical when working with multiple stakeholders or workflows. Implement tools like versioning, logging, and health checks to verify that data remains consistent and available. +Build reliable data collection and validation pipelines with real-time monitoring, automated reconciliation, and recovery procedures to maintain measurement accuracy in advertising systems. @@ -13 +13,27 @@ Data consistency and availability is critical when working with multiple stakeho -## Implementation guidance +**Desired outcome:** + + * Measurement events are collected with at-least-once delivery guarantees and deduplicated before processing. + + * Data quality checks run at each pipeline stage to catch anomalies before they propagate to reporting. + + * Failed measurement events are processed from dead-letter queues rather than discarded. + + + + +**Common anti-patterns:** + + * Collecting measurement events without at-least-once delivery guarantees, so events are lost on failure. + + * Discarding failed events instead of routing them to a dead-letter queue for reprocessing. + + * Running data quality checks only at the end, so anomalies propagate to reporting before detection. + + * No reconciliation against third-party provider totals, so collection drift accumulates undetected. + + + + +**Benefits of establishing this best practice:** + + * At-least-once collection with deduplication supports measurement completeness without double counting. @@ -15 +41 @@ Data consistency and availability is critical when working with multiple stakeho - * Implement versioning for collaborative datasets and schemas. + * Stage-by-stage data quality checks catch anomalies before they reach reporting. @@ -17 +43 @@ Data consistency and availability is critical when working with multiple stakeho - * Use transaction logs for tracking privacy computation state. + * Dead-letter queue processing recovers failed events rather than losing them. @@ -19 +44,0 @@ Data consistency and availability is critical when working with multiple stakeho - * Configure cross-Region replication for critical data stores. @@ -21 +45,0 @@ Data consistency and availability is critical when working with multiple stakeho - * Implement idempotency for matching operations. @@ -23 +46,0 @@ Data consistency and availability is critical when working with multiple stakeho - * Set up health checks for collaboration service endpoints. @@ -25 +48 @@ Data consistency and availability is critical when working with multiple stakeho - * Use read replicas for high-availability data access. +**Level of risk exposed if this best practice is not established:** High @@ -27 +50 @@ Data consistency and availability is critical when working with multiple stakeho - * Configure automated rollback procedures for failed operations. +## Implementation guidance @@ -28,0 +52 @@ Data consistency and availability is critical when working with multiple stakeho +The longer an error remains, the higher its corresponding cost for correction. Once an event enters the Kinesis pipeline, it spreads to aggregation, reporting, and billing. Fixing it there requires a full AWS Glue ETL reprocessing run and possible advertiser billing adjustments. Validate close to collection to help avoid such steadily rising costs. @@ -29,0 +54 @@ Data consistency and availability is critical when working with multiple stakeho +At-least-once delivery accepts duplicates in exchange for never losing an event. Pair it with DynamoDB-backed deduplication to remove the duplicates the guarantee allows. Together, they give completeness without double counting. @@ -30,0 +56 @@ Data consistency and availability is critical when working with multiple stakeho +Accept events to S3 and route validation failures to an SQS dead-letter queue. Store the original events in an S3 bucket to preserve valid data when a client is misconfigured, and use a dead-letter queue to block invalid events from moving downstream undetected. @@ -32 +58 @@ Data consistency and availability is critical when working with multiple stakeho -## Key AWS services +A KCL consumer can fail without raising an alarm, and small persistent losses cause measurement drift over time. Reconciliation against third-party totals eventually exposes the drift, but by then some data might be unrecoverable. Assign an owner to the reconciliation cadence who acts on its findings. Add stage-by-stage data quality checks so that owner detects anomalies earlier than a final check would. @@ -34 +60 @@ Data consistency and availability is critical when working with multiple stakeho - * Amazon DynamoDB +### Implementation steps @@ -36 +62 @@ Data consistency and availability is critical when working with multiple stakeho - * Amazon S3 + 1. **Deploy Kinesis Data Streams for measurement event collection:** Create Kinesis shards sized for your peak measurement event throughput (each shard handles 1,000 records/second). Configure Kinesis Consumer Library (KCL) with automatic checkpoint management. Set ShardIteratorType to TRIM_HORIZON during recovery scenarios so that no events are missed. @@ -38 +64 @@ Data consistency and availability is critical when working with multiple stakeho - * AWS Lambda + 2. **Implement dead-letter queue processing for all measurement consumers:** Configure DLQ on all SQS queues and Kinesis stream consumers. Set DLQ depth alarms at 100 messages (immediate investigation required). Deploy Lambda functions that process DLQ messages on a 15-minute schedule, attempting reprocessing with exponential backoff. Generate reports on DLQ processing for data loss tracking. @@ -40 +66 @@ Data consistency and availability is critical when working with multiple stakeho - * Amazon CloudWatch + 3. **Run automated reconciliation between measurement sources:** Deploy a daily AWS Glue job that compares measurement totals from your collection pipeline against third-party verification provider reports. Flag discrepancies exceeding 2% for investigation. Store reconciliation results in S3 and publish summary metrics to CloudWatch for trend analysis. @@ -47 +73,27 @@ Data consistency and availability is critical when working with multiple stakeho - * [Guidance for Maximum Data Availability Architecture on AWS](https://aws.amazon.com/solutions/guidance/maximum-data-availability-architecture-on-aws/) +**Related best practices:** + + * [ADVREL08-BP01 Implement redundant ad verification systems with automated failover mechanisms](./advrel08-bp01.html) + + * [ADVREL02-BP02 Implement a caching strategy](./advrel02-bp02.html) + + + + +**Related documents:** + + * [Avoiding insurmountable queue backlogs - Amazon Builders' Library](https://aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs/) + + * [Making retries safe with idempotent APIs - Amazon Builders' Library](https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-apis/) + + + + +**Related services:** + + * [Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/) + + * [Amazon SQS](https://aws.amazon.com/sqs/) + + * [Amazon S3](https://aws.amazon.com/s3/) + + * [AWS Glue](https://aws.amazon.com/glue/) @@ -49 +101 @@ Data consistency and availability is critical when working with multiple stakeho - * [CAP theorem](https://docs.aws.amazon.com/whitepapers/latest/availability-and-beyond-improving-resilience/cap-theorem.html) + * [Amazon EMR](https://aws.amazon.com/emr/) @@ -60 +112 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -ADVREL08-BP01 Design resilient architectures with privacy-preserving fault tolerance +ADVREL08-BP01 Implement redundant ad verification systems with automated failover mechanisms @@ -62 +114 @@ ADVREL08-BP01 Design resilient architectures with privacy-preserving fault toler -ADVREL08-BP03 Implement secure and privacy-preserving recovery mechanisms for collaboration workloads +Prebid Server resilience