AWS wellarchitected: Rewrite ADVPERF03-BP02 object storage best practice for ad data
Summary
Expanded the performance best practice for using object storage (Amazon S3) to store and analyze advertising data, adding desired outcomes, anti-patterns, benefits, implementation steps, and related resources. Includes a step to use AWS CloudTrail for S3 API activity logging and security auditing.
Security assessment
The change is primarily a performance best practice rewrite, but it includes a step documenting AWS CloudTrail logging for S3 API activity for compliance and security auditing. This is security-adjacent logging/audit guidance, not a fix for a specific vulnerability.
Evidence
+ 5. **Monitor S3 access and performance:** Use AWS CloudTrail to log S3 API activity for compliance and security auditing. Use [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) S3 storage metrics to track request counts, error rates, and first-byte latency by bucket and prefix. Enable S3 Storage Lens for organization-wide visibility into storage utilization, activity trends, and cost optimization opportunities across all your advertising data buckets.
Diff
diff --git a/wellarchitected/latest/video-streaming-advertising-lens/advperf03-bp02.md b/wellarchitected/latest/video-streaming-advertising-lens/advperf03-bp02.md index 96f0ca5..ca2b1b1 100644 --- a//wellarchitected/latest/video-streaming-advertising-lens/advperf03-bp02.md +++ b//wellarchitected/latest/video-streaming-advertising-lens/advperf03-bp02.md @@ -9 +9 @@ Implementation guidanceResources -# ADVPERF03-BP02 Use object storage to store and analyze raw data from ad servers, DSPs, and DMP +# ADVPERF03-BP02 Use object storage to store and analyze raw data from ad servers, DSPs, and DMPs @@ -11 +11,34 @@ Implementation guidanceResources -Object storage can be used to store massive amounts of data while balancing cost and performance. Customers can use object storage services to build data lakes and analyze this data to uncover valuable insights and achieve business goals. +[Amazon S3](https://aws.amazon.com/s3/) can store massive amounts of advertising data while balancing cost and performance. Use object storage to build data lakes and analyze this data, from real-time bid analytics to historical attribution modeling. + +**Desired outcome:** + + * Advertising data from raw event logs and campaign assets to ML training datasets and analytics pipelines is stored in Amazon S3 with the right storage class, partitioning strategy, and access configuration for each workload type. + + * Storage costs scale efficiently with data volume, and performance-critical workloads are not bottlenecked by storage tier mismatches. + + * S3 Intelligent-Tiering automatically optimizes storage costs for advertising data with variable or unpredictable access patterns. + + + + +**Common anti-patterns:** + + * Storing all advertising data under a single S3 prefix regardless of access frequency or performance requirements, which creates request-rate bottlenecks on high-throughput ingestion paths. + + * Manually managing storage class transitions for large advertising data lakes instead of using automated tiering, so teams pay either unnecessary storage cost or operational overhead. + + * Not using S3 Transfer Acceleration for long-distance data ingestion from distributed ad servers and DSPs. + + + + +**Benefits of establishing this best practice:** + + * Amazon S3 provides virtually unlimited, durable object storage with a tiering model you can use to optimize cost without sacrificing availability. + + * For advertising workloads that generate large volumes of event data, campaign assets, and model training inputs, matching each data category to the right storage class reduces storage spend while keeping performance-critical paths fast. + + + + +**Level of risk exposed if this best practice is not established:** High @@ -15 +48,5 @@ Object storage can be used to store massive amounts of data while balancing cost -[Amazon S3](https://aws.amazon.com/s3/) is a highly scalable and durable object storage service that can store and protect any amount of data for a range of use cases. It is ideal for storing and serving static content, such as images, videos, and other media assets used in advertising campaigns. Amazon S3 also supports data lakes, which you can use to store and analyze vast amounts of raw data from various sources, including ad servers, demand-side platforms (DSPs), and data management platforms (DMPs). +Advertising platforms generate multiple categories of data with widely different access patterns and performance requirements. Bid logs and impression events are written continuously. They are read heavily in the hours immediately after generation for pacing and frequency capping, then accessed infrequently for historical analysis. Campaign creative assets are written once during campaign setup and read repeatedly during active delivery, then rarely accessed afterward. ML training datasets are accessed intensively during model training runs, then idle between training cycles. + +[Amazon S3's tiering model](https://aws.amazon.com/s3/storage-classes/) provides a cost tier for each access pattern. Use S3 Intelligent-Tiering for advertising data lakes where access patterns change over campaign lifecycles. Hot content automatically transitions to lower-cost tiers as the campaign completes, with no lifecycle rules to manage. + +Amazon S3 request throughput scales with each prefix. A single prefix supporting millions of bid log writes per day can become a throughput bottleneck. Distribute objects across multiple prefixes (by date, campaign ID, or data source) to spread request load and maximize available throughput. @@ -17 +54 @@ Object storage can be used to store massive amounts of data while balancing cost - * **[Amazon S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/):** A powerful storage class for performance-critical applications, including advertising model training. Its low latency, high throughput, and cost efficiency makes it an ideal choice for real-time ad placement, machine learning for ad personalization, and interactive analytics. +Amazon S3 Transfer Acceleration routes uploads through the nearest AWS edge location and onto the AWS backbone network. This reduces the variability and latency of long-distance data transfers for advertising platforms with globally distributed ad servers or DSP integrations. Use it to collect bid logs from European or Asia-Pacific ad infrastructure into a primary US-based analytics data lake. @@ -19 +56 @@ Object storage can be used to store massive amounts of data while balancing cost - * **Data partitioning:** Use multiple prefixes to partition your data, which distributes the load and improves performance. For example, instead of storing all objects under a single prefix, use multiple prefixes like `s3://bucket-name/prefix1/` and `s3://bucket-name/prefix2/`. +### Implementation steps @@ -21 +58 @@ Object storage can be used to store massive amounts of data while balancing cost - * **Data transfer:** Use Amazon S3 Transfer Acceleration to speed up data transfers over long distances, improving the performance of data ingestion and distribution processes. + 1. **Partition advertising data across multiple S3 prefixes:** Design prefix hierarchies that distribute request load. For bid log data, use a prefix structure such as bid-logs/year=YYYY/month=MM/day=DD/hour=HH/ to partition by time. For campaign creative assets, partition by campaign ID. This enables partition pruning in [Amazon Athena](https://aws.amazon.com/athena/) queries and limits request rate throttling on a single prefix. @@ -23 +60 @@ Object storage can be used to store massive amounts of data while balancing cost - * **Monitoring and auditing:** Use AWS CloudTrail and Amazon CloudWatch to monitor S3 access and performance metrics. + 2. **Apply S3 Intelligent-Tiering to advertising data with variable access patterns:** Enable S3 Intelligent-Tiering on buckets containing campaign data, audience segments, and historical bid logs where access frequency changes over campaign lifecycles. Configure the optional Archive Access tier (objects not accessed for 90 days) and Deep Archive Access tier (objects not accessed for 180 days) for cost-optimized storage of completed campaign data. @@ -25 +62,5 @@ Object storage can be used to store massive amounts of data while balancing cost - * **Storage tiering and class:** Each object in Amazon S3 has a [storage class](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) associated with it. Choosing a storage class designed for your use case lets you optimize storage costs, performance, and availability for your objects. Use the S3 Intelligent-Tiering storage class, which is designed to optimize storage costs by automatically moving data to the most cost-effective access tier when access patterns change, without operational overhead or impact on performance. S3 Intelligent-Tiering monitors access patterns and automatically moves objects that have not been accessed to lower-cost access tier. + 3. **Use S3 Express One Zone for performance-critical advertising data:** For model artifacts, temporary creative assets, and intermediate audience processing data that require single-digit-millisecond access from your compute, use S3 Express One Zone directory buckets colocated in the same Availability Zone as your EC2 or ECS workloads (for more detail, see ADVPERF03-BP04). + + 4. **Enable S3 Transfer Acceleration for distributed data ingestion:** For advertising platforms with ad servers, DSPs, or data collectors operating in regions distant from your primary analytics bucket, enable S3 Transfer Acceleration on the destination bucket. Configure your data collection agents to use the <bucket>.s3-accelerate.amazonaws.com endpoint. Measure transfer latency improvement in your specific geographic configuration. Acceleration provides the most benefit for cross-continental transfers. + + 5. **Monitor S3 access and performance:** Use AWS CloudTrail to log S3 API activity for compliance and security auditing. Use [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) S3 storage metrics to track request counts, error rates, and first-byte latency by bucket and prefix. Enable S3 Storage Lens for organization-wide visibility into storage utilization, activity trends, and cost optimization opportunities across all your advertising data buckets. @@ -32 +73,4 @@ Object storage can be used to store massive amounts of data while balancing cost - * [Getting started with S3 Express One Zone](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-getting-started.html) +**Related best practices:** + + * [ADVPERF03-BP04 Use single-zone high-performance object storage for latency-sensitive ad serving and model artifact retrieval](./advperf03-bp04.html) + @@ -34 +77,0 @@ Object storage can be used to store massive amounts of data while balancing cost - * [Setting an S3 Lifecycle configuration on a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-set-lifecycle-configuration-intro.html) @@ -36 +78,0 @@ Object storage can be used to store massive amounts of data while balancing cost - * [Protecting data with server-side encryption](Users/jblatch/Downloads/%E2%80%A2%20https:/docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html) @@ -38 +80,3 @@ Object storage can be used to store massive amounts of data while balancing cost - * [Monitoring metrics with Amazon CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudwatch-monitoring.html) +**Related documents:** + + * [Getting started with S3 Express One Zone](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-one-zone.html) @@ -44,0 +89,11 @@ Object storage can be used to store massive amounts of data while balancing cost +**Related services:** + + * [Amazon S3](https://aws.amazon.com/s3/) + + * [Amazon S3 Intelligent-Tiering](https://aws.amazon.com/s3/storage-classes/intelligent-tiering/) + + * [Amazon Athena](https://aws.amazon.com/athena/) + + + +