AWS eks documentation change
Summary
Expanded documentation for monitoring AI/ML workloads on EKS with additional metrics examples, code samples for GPU memory tracking, inference latency, token throughput, and checkpoint restore monitoring using CloudWatch and Prometheus/Grafana
Security assessment
Changes focus on operational observability improvements rather than security vulnerabilities. Added metrics help prevent OOM errors and optimize performance but don't address authentication, vulnerabilities, or data protection. No evidence of patching security flaws or describing security controls.
Diff
diff --git a/eks/latest/best-practices/aiml-observability.md b/eks/latest/best-practices/aiml-observability.md index dde8edc7f..3dd992f1d 100644 --- a//eks/latest/best-practices/aiml-observability.md +++ b//eks/latest/best-practices/aiml-observability.md @@ -27 +27,14 @@ Optimizing Amazon EKS clusters for AI/ML workloads presents unique monitoring ch -Frameworks such as [vLLM](https://github.com/vllm-project/vllm), [Ray](https://github.com/ray-project/ray), and [Hugging Face TGI](https://huggingface.co/docs/text-generation-inference/en/index), etc., provide these native metrics out of the box, enabling seamless integration for monitoring distributed inference workloads without additional custom setup. +#### Tools and frameworks + +Certain tools and frameworks offer native, out-of-the-box metrics for monitoring AI/ML workloads, enabling easier integration without additional custom setup. These focus on performance aspects such as latency, throughput, and token generation, which are critical for inference serving and benchmarking. Examples include: + + * **vLLM** : A high-throughput serving engine for large language models (LLMs) that provides native metrics such as request latency and memory usage. + + * **Ray** : A distributed computing framework that emits metrics for scalable AI workloads, including task execution times and resource utilization. + + * **Hugging Face Text Generation Inference (TGI)** : A toolkit for deploying and serving LLMs, with built-in metrics for inference performance. + + * **NVIDIA genai-perf** : A command-line tool for benchmarking generative AI models, measuring throughput, latency, and LLM-specific metrics, such as requests completed in specific time intervals. + + + @@ -51 +64 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **CPU, Memory, and GPU Usage** — Monitoring these metrics for ML workloads allows you to ensure the allocated resources are sufficient and identify opportunities for optimization. For example, monitor the CPU usage of your training pods to ensure they have enough compute resources. + * **CPU, Memory, GPU and GPU Memory Usage** — Monitoring these metrics for ML workloads allows you to ensure the allocated resources are sufficient and identify opportunities for optimization. For example, tracking metrics like `gpu_memory_usage_bytes`, you can identify memory consumption patterns, detect peak usage, and calculate percentiles such as the 95th percentile (P95) to understand the highest memory demands during training. This helps in optimizing your models and infrastructure to avoid OOM errors and reduce costs. @@ -62 +75 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Model Performance Metrics:** +**Model Performance Metrics** : @@ -73 +86 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Data Quality and Drift Metrics:** +**Data Quality and Drift Metrics** : @@ -82 +95,7 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Latency and Throughput Metrics:** +**Latency and Throughput Metrics** : + + * **End-to-End Latency of ML Training Pipelines** — Monitor the time it takes for data to flow through the entire training pipeline. For example, measure total time from data ingestion to model update. + + * **Training Throughput and Processing Rate** — Track the volume of data processed during training to ensure efficiency. For example, monitor positive and negative samples processed per second. + + * **Checkpoint Restore Latency** – Monitor the time taken to load a saved model checkpoint from a storage location (S3, EFS, FSx etc) back to GPU/CPU memory when resuming a job, recovering from failure, or initializing. This metric directly impacts job recovery time, cold start performance, and overall efficiency of interference pipelines. In auto-scaling inference services, slow checkpoint loading can cause cold start delays and a degrading user experience. These related metrics are also commonly used to improve model checkpointing: checkpoint downtime latency, model deserialization time, checkpoint size, and checkpoint restore failure count. @@ -84 +103 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **End-to-End Latency of ML Pipelines** — Monitor the time it takes for data to flow through the entire training pipeline. For example, measure total time from data ingestion to model update. + * **Inference Request Duration** – Monitor the time it takes to complete an inference request. This is the time from initial request received to completed response from the model. @@ -86 +105 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Throughput and Processing Rate** — Track the volume of data processed during training to ensure efficiency. For example, monitor positive and negative samples processed per second. + * **Token Throughput** \- Monitor token processing time to gauge model performance and optimize scaling decisions. Slow processing can indicate inefficiencies or underutilized resources, impacting cost-effectiveness. Tracking metrics like tokens in per second and tokens out per second, alongside processing time, can help identify performance bottlenecks, slowdowns, and cost drivers. @@ -102 +121 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Kubernetes and EKS Specific Metrics:** +**Kubernetes and EKS Specific Metrics** : @@ -114,0 +134,2 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +In subsequent topics, we demonstrate gathering data for a few of the metrics mentioned above. We will provide examples with the two AWS recommended approaches: [AWS-native CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) and open-source [Amazon Managed Prometheus](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-getting-started.html) with [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html). You would choose one of these solutions based on your overall observability needs. See [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html) for the complete list of Container Insights metrics. + @@ -138,0 +160,250 @@ K/V (Key/Value) cache can be a powerful optimization technique for inference lat +In subsequent topics, we demonstrate gathering data for a few of the metrics mentioned above. We will provide examples with the two AWS recommended approaches: [AWS-native CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) and open-source [Amazon Managed Prometheus](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-getting-started.html) with [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html). You would choose one of these solutions based on your overall observability needs. See [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html) for the complete list of Container Insights metrics. + +### Tracking GPU Memory Usage + +As discussed in the Consider Monitoring Core Training & Fine-Tuning Metrics topic, GPU memory usage is essential to prevent out-of-memory (OOM) errors and ensure efficient resource utilization. The following examples show how to instrument your training application to expose a custom histogram metric, `gpu_memory_usage_bytes`, and calculate the P95 memory usage to identify peak consumption. Be sure to test with a sample training job (e.g., fine-tuning a transformer model) in a staging environment. + +**AWS-Native CloudWatch Container Insights Example** + +This sample demonstrates how to instrument your training application to expose `gpu_memory_usage_bytes` as a histogram using the AWS-native approach. Note that your AI/ML container must be configured to emit structured logs in CloudWatch [Embedded Metrics Format (EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html) format. CloudWatch logs parses EMF and publishes the metrics. Use [aws_embedded_metrics](https://github.com/awslabs/aws-embedded-metrics-python) in your training application to send structured logs in EMF format to CloudWatch Logs, which extracts GPU metrics. + + + from aws_embedded_metrics import metric_scope + import torch + import numpy as np + + memory_usage = [] + + @metric_scope + def log_gpu_memory(metrics): + # Record current GPU memory usage + mem = torch.cuda.memory_allocated() + memory_usage.append(mem) + + # Log as histogram metric + metrics.set_namespace("MLTraining/GPUMemory") + metrics.put_metric("gpu_memory_usage_bytes", mem, "Bytes", "Histogram") + + # Calculate and log P95 if we have enough data points + if len(memory_usage) >= 10: + p95 = np.percentile(memory_usage, 95) + metrics.put_metric("gpu_memory_p95_bytes", p95, "Bytes") + print(f"Current memory: {mem} bytes, P95: {p95} bytes") + + # Example usage in training loop + for epoch in range(20): + # Your model training code would go here + log_gpu_memory() + +**Prometheus and Grafana Example** + +This sample demonstrates how to instrument your training application to expose `gpu_memory_usage_bytes`` as a histogram using the Prometheus client library in Python. + + + from prometheus_client import Histogram + from prometheus_client import start_http_server + import pynvml + + start_http_server(8080) + memory_usage = Histogram( + 'gpu_memory_usage_bytes', + 'GPU memory usage during training', + ['gpu_index'], + buckets=[1e9, 2e9, 4e9, 8e9, 16e9, 32e9] + ) + + # Function to get GPU memory usage + def get_gpu_memory_usage(): + if torch.cuda.is_available(): + # Get the current GPU device + device = torch.cuda.current_device() + + # Get memory usage in bytes + memory_allocated = torch.cuda.memory_allocated(device) + memory_reserved = torch.cuda.memory_reserved(device) + + # Total memory usage (allocated + reserved) + total_memory = memory_allocated + memory_reserved + + return device, total_memory + else: + return None, 0 + + # Get GPU memory usage + gpu_index, memory_used = get_gpu_memory_usage() + +### Track Inference Request Duration for Real-Time Online Inference + +As discussed in the Consider Monitoring Core Training & Fine-Tuning Metrics topic, low latency is critical for providing timely responses to users or other dependent systems. The following examples show how to track a custom histogram metric, `inference_request_duration_seconds`, exposed by your inference application. Calculate the 95th percentile (P95) latency to focus on worst-case scenarios, test with synthetic inference requests (e.g., via Locust) in a staging environment, and set alert thresholds (e.g., >500ms) to detect SLA violations. + +**AWS-Native CloudWatch Container Insights Example** + +This sample demonstrates how to create a custom histogram metric in your inference application for inference_request_duration_seconds using AWS CloudWatch Embedded Metric Format. + + + import boto3 + import time + from aws_embedded_metrics import metric_scope, MetricsLogger + + cloudwatch = boto3.client('cloudwatch') + + @metric_scope + def log_inference_duration(metrics: MetricsLogger, duration: float): + metrics.set_namespace("ML/Inference") + metrics.put_metric("inference_request_duration_seconds", duration, "Seconds", "Histogram") + metrics.set_property("Buckets", [0.1, 0.5, 1, 2, 5]) + + @metric_scope + def process_inference_request(metrics: MetricsLogger): + start_time = time.time() + + # Your inference processing code here + # For example: + # result = model.predict(input_data) + + duration = time.time() - start_time + log_inference_duration(metrics, duration) + + print(f"Inference request processed in {duration} seconds") + + # Example usage + process_inference_request() + +**Prometheus and Grafana Example** + +This sample demonstrates how to create a custom histogram metric in your inference application for inference_request_duration_seconds using the Prometheus client library in Python: + + + from prometheus_client import Histogram + from prometheus_client import start_http_server + import time + + start_http_server(8080) + request_duration = Histogram( + 'inference_request_duration_seconds', + 'Inference request latency', + buckets=[0.1, 0.5, 1, 2, 5] + ) + start_time = time.time() + # Process inference request + request_duration.observe(time.time() - start_time) + +In Grafana, use the query `histogram_quantile(0.95, sum(rate(inference_request_duration_seconds_bucket[5m])) by (le, pod))` to visualize P95 latency trends. To learn more, see [Prometheus Histogram Documentation](https://prometheus.io/docs/practices/histograms/) and [Prometheus Client Documentation](https://github.com/prometheus/client_python). + +### Track Token Throughput for Real-Time Online Inference + +As discussed in the Consider Monitoring Core Training & Fine-Tuning Metrics topic, we recommend monitoring token processing time to gauge model performance and optimize scaling decisions. The following examples show how to track a custom histogram metric, `token_processing_duration_seconds`, exposed by your inference application. Calculate the 95th percentile (P95) duration to analyze processing efficiency, test with simulated request loads (e.g., 100 to 1000 requests/second) in a non-production cluster, and adjust KEDA triggers to optimize scaling. + +**AWS-Native CloudWatch Container Insights Example** + +This sample demonstrates how to create a custom histogram metric in your inference application for token_processing_duration_seconds using AWS CloudWatch Embedded Metric Format. It uses dimensions (`set_dimension`) with a custom `get_duration_bucket` function to categorize durations into buckets (e.g., "⇐0.01", ">1"). + + + import boto3 + import time + from aws_embedded_metrics import metric_scope, MetricsLogger + + cloudwatch = boto3.client('cloudwatch') + + @metric_scope