AWS solutions documentation change
Summary
Updated solution guidance to shift from Amazon SageMaker to Amazon ECS for model inference, including architectural changes, revised workflow steps, and updated technical details across all pillars (Operational Excellence, Security, Reliability, etc.).
Security assessment
No security vulnerabilities or incidents are mentioned. Changes focus on architectural redesign (SageMaker → ECS) and technical implementation updates. Security section was simplified to IAM policies only, removing references to HTTPS encryption, KMS, and PrivateLink without indicating a security fix.
Diff
diff --git a/solutions/low-latency-high-throughput-model-inference-using-amazon-ecs/index.md b/solutions/low-latency-high-throughput-model-inference-using-amazon-ecs/index.md index 1dd7b83c7..fc201fab4 100644 --- a//solutions/low-latency-high-throughput-model-inference-using-amazon-ecs/index.md +++ b//solutions/low-latency-high-throughput-model-inference-using-amazon-ecs/index.md @@ -1,3 +1 @@ -# Guidance for Low-Latency, High Throughput Model Inference Using Amazon SageMaker - -[Open guide](https://aws-solutions-library-samples.github.io/advertising-marketing/low-latency-high-throughput-model-inference-using-amazon-sagemaker.html)[Go to sample code](https://github.com/aws-solutions-library-samples/guidance-for-low-latency-high-throughput-model-inference-using-amazon-sagemaker) +# Guidance for Low-Latency High-Throughput Model Inference Using Amazon ECS @@ -7,4 +5 @@ -This Guidance shows how to use Amazon SageMaker to support high-throughput model inferencing workloads like programmatic advertising and real-time bidding (RTB). For instance, your demand-side platform could use machine learning (ML) models to determine whether to place a bid for an advertising campaign and at what price. By using this Guidance, you can cost-effectively scale to millions of requests per second at a low latency. - -Note: Before beginning this Guidance, you will need to containerize your models. [SageMaker Model Training](/sagemaker/train/) provides a wide range of built-in algorithms and frameworks (such as for scikit-learn and XGBoost) you can use to train and tune your ML models. Alternatively, you can bring your own script. - +This Guidance demonstrates how to build a real-time machine learning (ML) inferencing solution on AWS that can serve millions of requests per second. By hosting your solution’s ML model on Amazon Elastic Container Service (Amazon ECS) and routing requests to the ML server using Network Load Balancer, you can achieve low latency and support high-throughput inference requirements commonly found in real-time and programmatic advertising. This Guidance provides an example of applying ML for ad request filtering and demonstrates how to build a client application that can simulate high-throughput OpenRTB-based requests to send to the ML inference server. @@ -17,7 +12 @@ These technical details feature an architecture diagram to illustrate how to eff -[Download the architecture diagram ](https://d1.awsstatic.com/solutions/guidance/architecture-diagrams/low-latency-high-throughput-model-inference-using-amazon-sagemaker.pdf) - - Step 1 - -A consumer application is deployed within a virtual private cloud (VPC) in your AWS account, using Amazon Virtual Private Cloud (Amazon VPC). This application can be hosted on Amazon Elastic Compute Cloud (Amazon EC2) instances or as containers running on either Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS). - -Step 2 +[Download the architecture diagram ](https://d1.awsstatic.com/solutions/guidance/architecture-diagrams/low-latency-high-throughput-model-inference-using-amazon-ecs.pdf) @@ -25 +14 @@ Step 2 -The consumer application connects to Amazon SageMaker Real-Time inference servers using VPC endpoints powered by AWS PrivateLink. This means that all API calls happen over the private network of AWS and not the public internet, minimizing the latency of the invocations and improving your security posture. + Step A @@ -27 +16 @@ The consumer application connects to Amazon SageMaker Real-Time inference server -Step 3 +Data scientists use Amazon SageMaker to experiment with, build, and train their ML model. Once the model is ready, it is saved in Amazon Simple Storage Service (Amazon S3). @@ -29 +18 @@ Step 3 -The inference requests are routed through a Network Load Balancer to the SageMaker real-time inference servers. These servers are hosted across multiple Availability Zones (AZs) within an Amazon EC2 Auto Scaling group. This allows the model inference infrastructure to be elastic and highly available. SageMaker real-time inferences provide a choice of Amazon EC2 instance types. These include Amazon EC2 Inf1 instances based on AWS Inferentia, high-performance machine learning (ML) inference chips designed and built by AWS, and GPU instances, such as Amazon EC2 G4dn. Multiple hosting options, including shadow testing and an inference recommendation feature in the managed service, reduce operational burden and accelerates time to value. +Step B @@ -31 +20 @@ The inference requests are routed through a Network Load Balancer to the SageMak -Step 4 +The trained model is read and loaded by the Amazon Elastic Container Service (Amazon ECS) model inference task. The model is hosted as a Thrift endpoint. Incoming requests, in OpenRTB format (for real-time bidding), are used for inference. @@ -33 +22 @@ Step 4 -Consumer applications and batch applications use Amazon Simple Storage Service (Amazon S3) to store and retrieve data and use it for offline ML training and experiments. Access to Amazon S3 from the VPC is secured through PrivateLink. +Step 1 @@ -35 +24 @@ Consumer applications and batch applications use Amazon Simple Storage Service ( -Step 5 +A publisher issues requests to a supply-side platform (SSP) auction server for an ad placement. @@ -37 +26 @@ Step 5 -Data scientists use SageMaker to experiment, build, and train the ML model. Once the model is ready, it is saved in Amazon S3 for the model inference task to load. Access to Amazon S3 from the VPC is again secured through PrivateLink. +Step 2 @@ -39 +28 @@ Data scientists use SageMaker to experiment, build, and train the ML model. Once -## Deploy with confidence +The auction server (a client application) is hosted as an Amazon ECS application within the SSP's virtual private cloud (VPC). The auction request issues a bid request based on the OpenRTB format. @@ -41 +30 @@ Data scientists use SageMaker to experiment, build, and train the ML model. Once -Everything you need to launch this Guidance in your account is right here. +Step 3 @@ -43 +32 @@ Everything you need to launch this Guidance in your account is right here. -### Let's make it happen +Network Load Balancer distributes the incoming requests to an Amazon Elastic Compute Cloud (Amazon EC2)-based Amazon ECS cluster that hosts the ad-filtering ML server. The purpose of the ad-filtering ML server is to infer the likelihood of a bid for every bid request, filtering the demand partners that need to be sent to the auction request, and optimizing the cost per bid. @@ -45 +34 @@ Everything you need to launch this Guidance in your account is right here. -Dive deep into the implementation guide for additional customization options and service configurations to tailor to your specific needs. +Step 4 @@ -47 +36 @@ Dive deep into the implementation guide for additional customization options and -[Open guide](https://aws-solutions-library-samples.github.io/advertising-marketing/low-latency-high-throughput-model-inference-using-amazon-sagemaker.html) [Go to sample code](https://github.com/aws-solutions-library-samples/guidance-for-low-latency-high-throughput-model-inference-using-amazon-sagemaker) +The ad-filtering ML server is hosted as a container within an Amazon EC2-based Amazon ECS cluster. An Amazon EC2 Auto Scaling group maintains the desired number of Amazon EC2 instances running across multiple Availability Zones (AZs) to maintain high availability. Amazon ECS deploys and maintains the desired capacity of the Amazon ECS tasks, hosting the ML container. Each task loads the ad-filtering model from an Amazon S3 bucket and hosts it as a Thrift protocol-based endpoint. This helps in low-latency-based communication, and multiple instances of the tasks support a high number of concurrent requests. @@ -55 +44 @@ Operational Excellence -Amazon CloudWatch aggregates logs and creates observability metrics and dashboards, providing visualizations to help you identify performance bottlenecks and troubleshoot requests. You can also set up CloudWatch alarms to identify trends that could be problematic and alert you before they impact your application or business. Additionally, you can use AWS CloudTrail, which keeps track of account activity, to enable governance and risk auditing, as well as facilitate the compliance of your AWS account. +Amazon CloudWatch monitors the performance of the Amazon ECS cluster (including CPU and memory) along with the incoming requests sent through Network Load Balancer. Your CloudWatch dashboard—created as part of an AWS CloudFormation script—provides a comprehensive view of the number of incoming requests and their associated latency. By using CloudWatch to visualize and analyze performance and latency, you can better identify any bottlenecks in your application. @@ -61 +50 @@ Security -The principle of least privilege is the industry best practice for reducing the surface area of security risks. AWS Identity and Access Management (IAM) policies use least-privilege access so that every policy is restrictive to the specific resource and operation. Additionally, to implement security in layers, this Guidance encrypts data in transit and transfers it over HTTPS, and AWS Key Management Service (AWS KMS) keys encrypt data at rest in Amazon S3 buckets. Finally, real-time bidding (RTB) applications access SageMaker endpoints and Amazon S3 only through PrivateLink, enhancing your security posture. +By scoping down all AWS Identity and Access Management (IAM) policies to the minimum permissions required for the services to function properly, you can limit unauthorized access to resources. @@ -67 +56 @@ Reliability -All the services used in this Guidance are serverless and can automatically scale horizontally based on workload demand. In the SageMaker inference endpoints, Amazon EC2 Auto Scaling groups launch instances across AZs to provide high availability. Additionally, Amazon S3 supports features like S3 Versioning, which helps you maintain data version control, prevent accidental deletions, and replicate data to the same or a different AWS Region. With the ability to preserve, retrieve, and restore every version of an object stored in Amazon S3, you can recover from unintended user actions and application failures. +The Amazon ECS cluster runs a service definition that maintains a desired capacity of EC2 instances. If one of the instances becomes unavailable, a new instance will automatically launch and be registered with the Amazon ECS cluster as a healthy target to receive incoming requests routed by Network Load Balancer. @@ -73 +62 @@ Performance Efficiency -AWS managed services offload infrastructure management and scaling from you so that you can focus on solving your business needs. In this Guidance, SageMaker manages the hosting of your model inference endpoints. It retrieves the models from Amazon S3 buckets at deployment time, then hosts the most optimal implementation runtime containers. By using its inference recommender and a load-testing tool, SageMaker can choose the optimal instance size based on throughput capacity and incurred latencies. This enables it to manage the scaling of the inference compute through load balancers and Amazon EC2 Auto Scaling groups. +Network Load Balancer, which communicates with Amazon ECS, supports low-millisecond latency and high throughput that are apt for this use case. @@ -79 +68 @@ Cost Optimization -This Guidance uses serverless technologies and managed services so that you only pay for the resources you consume. You can also select options to further reduce costs. For example, Amazon SageMaker Savings Plans offer a flexible, usage-based pricing model in exchange for a commitment to a consistent amount of usage. You can also store data cost-effectively by choosing from a range of Amazon S3 storage classes built for specific use cases and access patterns. For example, if you use Amazon S3 Intelligent-Tiering for data with changing, unknown, or unpredictable access patterns—such as data lakes, analytics, or new applications—it will automatically optimize costs by moving your data between tiers for frequent, infrequent, and rare access. Additionally, by securing traffic over a private network using PrivateLink, you can reduce data transfer fees. +Amazon EC2 Auto Scaling groups let you run your application at the desired capacity while providing dynamic support for scaling based on the load. Automatic scaling grows or reduces the infrastructure based on load and your scaling policy. This helps you control the costs associated with running your application. @@ -85 +74 @@ Sustainability -This Guidance uses serverless technologies that scale up and down to meet demand so that resources don’t consume energy while idle. Additionally, SageMaker endpoints use custom infrastructure that is optimal to the workload demands of model training and inference, helping you achieve more with fewer resources and a lower carbon footprint. +The Amazon EC2 -based Amazon ECS cluster lets you choose appropriate hardware types and configurations for specific workloads so that they run efficiently. As a result, you can maximize utilization and avoid overprovisioning resources. This Guidance is designed for low-latency and high-performance model inference workloads, so appropriate EC2 instance types are powered by AWS Graviton3. This service uses up to 60 percent less energy for the same performance as comparable EC2 instances, helping you reduce your carbon footprint.