AWS eks documentation change
Summary
Updated GPU monitoring guidance to emphasize power usage over utilization metrics, restructured training metrics sections with detailed subcategories, and enhanced observability implementation recommendations.
Security assessment
The changes focus exclusively on performance monitoring, cost optimization, and technical implementation of GPU metrics for AI/ML workloads. There are no references to security vulnerabilities, access controls, encryption, or compliance requirements. The modifications improve operational best practices without addressing security concerns.
Diff
diff --git a/eks/latest/best-practices/aiml-observability.md b/eks/latest/best-practices/aiml-observability.md index 9b0f00625..b8a8fe3ab 100644 --- a//eks/latest/best-practices/aiml-observability.md +++ b//eks/latest/best-practices/aiml-observability.md @@ -11 +11 @@ Monitoring and ObservabilityObservability and Metrics -### Target high GPU utilization +### GPU Metrics Explained @@ -13 +13 @@ Monitoring and ObservabilityObservability and Metrics -Underutilized GPUs indicate that the allocated GPU resources are not being fully leveraged by the workloads, leading to wasted compute capacity. For AI/ML workloads on Amazon EKS, we recommend monitoring GPU utilization to target high GPU usage and optimize resource efficiency. Underutilized GPUs waste compute capacity and increase costs, while over-scheduling can lead to contention and performance degradation. +The GPU Utilization metric shows whether the GPU ran any work during the sample window. This metric captures the percentage of time the GPU executed at least one instruction, but it does not reveal how efficiently the GPU used its hardware. A GPU contains multiple Streaming Multiprocessors (SMs), which are the parallel processing units that execute instructions. A 100% utilization reading can mean the GPU ran heavy parallel workloads across all its SMs, or it can mean a single small instruction activated the GPU over the sample period. To understand actual utilization, you need to examine GPU metrics at multiple levels of the hardware architecture. Each Streaming Multiprocessor is built from different core types, and each layer exposes different performance characteristics. Top-level metrics (GPU Utilization, Memory Utilization, GPU Power, and GPU Temperature, visible through nvidia-smi) show whether the device is active. Deeper metrics (SM utilization, SM Activity, and tensor core usage) reveal how efficiently the GPU uses its resources. @@ -15 +15 @@ Underutilized GPUs indicate that the allocated GPU resources are not being fully -We recommend setting up [Cloudwatch Container Insights on Amazon EKS](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) to identify specific pods, nodes, or workloads with low GPU utilization [metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html). It is easily integrated with Amazon EKS, enabling you to monitor GPU utilization and adjust pod scheduling or instance types if utilization falls below target levels. Alternatively, if this does not meet your specific requirements (e.g., advanced visualization), consider using NVIDIA’s DCGM-Exporter alongside Prometheus and Grafana for Kubernetes-native monitoring. Both approaches provide insights into GPU metrics, enabling you to adjust pod scheduling or instance types if utilization falls below target levels. Check [NVIDIA metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-NVIDIA-GPU.html) like `nvidia_smi_utilization_gpu` (GPU compute usage) and `nvidia_smi_utilization_memory` (GPU memory usage) via DCGM-Exporter or CloudWatch. Look for trends, such as consistently low utilization during certain hours or for specific jobs. +### Target high GPU power usage @@ -17 +17,5 @@ We recommend setting up [Cloudwatch Container Insights on Amazon EKS](https://do -Static resource limits in Kubernetes (e.g., CPU, memory, and GPU counts) can lead to over-provisioning or underutilization, particularly for dynamic AI/ML workloads like inference. We recommend analyzing utilization trends and consolidate workloads onto fewer GPUs, ensuring each GPU is fully utilized before allocating new ones. If GPUs are underutilized, consider the following strategies to optimize scheduling and sharing. To learn more, see the [EKS Compute and Autoscaling](https://docs.aws.amazon.com/eks/latest/best-practices/aiml-compute.html) best practices for details. +Underutilized GPUs waste compute capacity and increase costs because workloads fail to engage all GPU components simultaneously. For AI/ML workloads on Amazon EKS, track GPU power usage as a proxy to identify actual GPU activity. GPU Utilization reports the percentage of time the GPU executes any kernel, but it does not reveal whether the Streaming Multiprocessors, memory controllers, and tensor cores are all active at the same time. Power usage exposes this gap because fully engaged hardware draws significantly more power than hardware running lightweight kernels or sitting idle between tasks. Compare power draw against the GPU’s thermal design power (TDP) to spot underutilization, then investigate whether your workload is bottlenecked by CPU preprocessing, network I/O, or inefficient batch sizes. + +Set up CloudWatch Container Insights on Amazon EKS to identify pods, nodes, or workloads with low GPU power consumption. This tool integrates directly with Amazon EKS and allows you to monitor GPU power consumption and adjust pod scheduling or instance types when power usage falls below your target levels. If you need advanced visualization or custom dashboards, use NVIDIA’s DCGM-Exporter with Prometheus and Grafana for Kubernetes-native monitoring. Both approaches surface key NVIDIA metrics like `nvidia_smi_power_draw` (GPU power consumption) and `nvidia_smi_temperature_gpu` (GPU temperature). For a list of metrics, explore <https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-NVIDIA-GPU.htm>. Look for patterns such as consistently low power usage during specific hours or for particular jobs. These trends help you identify where to consolidate workloads or adjust resource allocation. + +Static resource limits in Kubernetes (such as CPU, memory, and GPU counts) often lead to over-provisioning or underutilization, especially for dynamic AI/ML workloads like inference where demand fluctuates. Analyze your utilization trends and consolidate workloads onto fewer GPUs. Ensure each GPU reaches full utilization before you allocate additional ones. This approach reduces waste and lowers costs. For detailed guidance on optimizing scheduling and sharing strategies, see the [EKS Compute and Autoscaling best practices](https://docs.aws.amazon.com/eks/latest/best-practices/aiml-compute.html) @@ -23 +27 @@ Static resource limits in Kubernetes (e.g., CPU, memory, and GPU counts) can lea -Modern AI/ML services operate at the intersection of infrastructure, modeling, and application logic. Platform engineers manage the infrastructure, observability stack and ensure metrics are collected, stored and visualized. AI/ML Engineers define model specific metrics and focus on performance under varying load and distribution. Application developers consume api, route requests and track service-level metrics and user interactions. Success depends on establishing unified observability practices across environments that give all stakeholders visibility into system health and performance. +Modern AI/ML services require coordination across infrastructure, modeling, and application logic. Platform engineers manage the infrastructure and observability stack. They collect, store, and visualize metrics. AI/ML engineers define model-specific metrics and monitor performance under varying load and data distribution. Application developers consume APIs, route requests, and track service-level metrics and user interactions. Without unified observability practices, these teams work in silos and miss critical signals about system health and performance. Establishing shared visibility across environments ensures all stakeholders can detect issues early and maintain reliable service. @@ -25 +29 @@ Modern AI/ML services operate at the intersection of infrastructure, modeling, a -Optimizing Amazon EKS clusters for AI/ML workloads presents unique monitoring challenges, particularly around GPU memory management. Without proper monitoring, organizations often face out-of-memory (OOM) errors, resource inefficiencies, and unnecessary costs. For EKS customers, effective monitoring ensures better performance, resilience, and lower costs. A holistic approach that combines granular GPU monitoring using [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html) (e.g., GPU memory used, free GPU memory), monitoring and optimizing inference serving for distributed workload insights with their native metrics using frameworks like [Ray](https://docs.ray.io/en/latest/serve/monitoring.html), [vLLM](https://docs.vllm.ai/en/v0.8.5/design/v1/metrics.html), and application level insights for custom metrics. +Optimizing Amazon EKS clusters for AI/ML workloads presents unique monitoring challenges, especially around GPU memory management. Without proper monitoring, organizations face out-of-memory (OOM) errors, resource inefficiencies, and unnecessary costs. Effective monitoring ensures better performance, resilience, and lower costs for EKS customers. Use a holistic approach that combines three monitoring layers. First, monitor granular GPU metrics using [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html) to track GPU power usage, GPU temperature, SM activity, SM occupancy, and XID errors. Second, monitor inference serving frameworks like [Ray](https://docs.ray.io/en/latest/serve/monitoring.html) and [vLLM](https://docs.vllm.ai/en/v0.8.5/design/v1/metrics.html) to gain distributed workload insights through their native metrics. Third, collect application-level insights to track custom metrics specific to your workload. This layered approach gives you visibility from hardware utilization through application performance. @@ -29 +33 @@ Optimizing Amazon EKS clusters for AI/ML workloads presents unique monitoring ch -Certain tools and frameworks offer native, out-of-the-box metrics for monitoring AI/ML workloads, enabling easier integration without additional custom setup. These focus on performance aspects such as latency, throughput, and token generation, which are critical for inference serving and benchmarking. Examples include: +Several tools and frameworks provide native, out-of-the-box metrics for monitoring AI/ML workloads. These built-in metrics eliminate the need for custom instrumentation and reduce setup time. The metrics focus on performance aspects such as latency, throughput, and token generation, which are critical for inference serving and benchmarking. Using native metrics allows you to start monitoring immediately without building custom collection pipelines. @@ -48 +52 @@ We recommend implementing any additional observability mechanisms in one of the -Once onboarded to Container Insights, CloudWatch automatically detects NVIDIA GPUs in your environment, collects the critical health and performance metrics of your NVIDIA GPUs as CloudWatch metrics and makes them available on curated out-of-the-box dashboards. Additionally, [Ray](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/aws.html) and [vLLM](https://docs.vllm.ai/en/latest/) can be integrated with CloudWatch using the [Unified CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/UseCloudWatchUnifiedAgent.html) to send metrics. This simplifies observability in EKS environments, allowing teams to focus on performance tuning and cost optimization. +After you install Container Insights, CloudWatch automatically detects NVIDIA GPUs in your environment and collects critical health and performance metrics. These metrics appear on curated out-of-the-box dashboards. You can also integrate [Ray](https://docs.ray.io/en/latest/cluster/vms/user-guides/launching-clusters/aws.html) and [vLLM](https://docs.vllm.ai/en/latest/) with CloudWatch using the [Unified CloudWatch Agent](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/UseCloudWatchUnifiedAgent.html) to send their native metrics. This unified approach simplifies observability in EKS environments and lets teams focus on performance tuning and cost optimization instead of building monitoring infrastructure. @@ -50 +54 @@ Once onboarded to Container Insights, CloudWatch automatically detects NVIDIA GP -For more information, to view a complete list of metrics available, see [Amazon EKS and Kubernetes Container Insights metrics.](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html#Container-Insights-metrics-EKS-GPU) Refer to [Gain operational insights for NVIDIA GPU workloads using Amazon CloudWatch Container Insights](https://aws.amazon.com/blogs/mt/gain-operational-insights-for-nvidia-gpu-workloads-using-amazon-cloudwatch-container-insights/) and [Optimizing AI responsiveness: A practical guide to Amazon Bedrock latency-optimized inference](https://aws.amazon.com/blogs/machine-learning/optimizing-ai-responsiveness-a-practical-guide-to-amazon-bedrock-latency-optimized-inference/) for detailed implementations. +For a complete list of available metrics, see [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-EKS.html#Container-Insights-metrics-EKS-GPU). For step-by-step guidance on implementing GPU monitoring, refer to [Gain operational insights for NVIDIA GPU workloads using Amazon CloudWatch Container Insights](https://aws.amazon.com/blogs/mt/gain-operational-insights-for-nvidia-gpu-workloads-using-amazon-cloudwatch-container-insights/). For practical examples of optimizing inference latency, see [Optimizing AI responsiveness: A practical guide to Amazon Bedrock latency-optimized inference](https://aws.amazon.com/blogs/machine-learning/optimizing-ai-responsiveness-a-practical-guide-to-amazon-bedrock-latency-optimized-inference/). @@ -52 +56 @@ For more information, to view a complete list of metrics available, see [Amazon -**Managed Prometheus and Grafana** If your organization is comfortable with open-source tools and customized dashboards, we recommend deploying Prometheus with the [NVIDIA DCGM-Exporter](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/k8s/containers/dcgm-exporter) and Grafana for Kubernetes-native monitoring with advanced visualizations. Prometheus scrapes metrics, and Grafana creates advanced visualizations. +**Managed Prometheus and Grafana** If your organization needs customized dashboards and advanced visualization capabilities, deploy Prometheus with the [NVIDIA DCGM-Exporter](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/k8s/containers/dcgm-exporter) and Grafana for Kubernetes-native monitoring. Prometheus scrapes and stores GPU metrics from the DCGM-Exporter, while Grafana provides flexible visualization and alerting capabilities. This approach gives you more control over dashboard design and metric retention compared to CloudWatch Container Insights. @@ -54 +58 @@ For more information, to view a complete list of metrics available, see [Amazon -Additionally, you can use open source frameworks like [Ray and vLLM](https://awslabs.github.io/ai-on-eks/docs/blueprints/inference/GPUs/vLLM-rayserve) to export metrics to Prometheus (which can be visualized using Grafana), or you can [Connect to an AWS X-Ray data source](https://docs.aws.amazon.com/grafana/latest/userguide/x-ray-data-source.html), then build dashboards to look at analytics, insights, or traces. +You can extend this monitoring stack by integrating open source frameworks like Ray and vLLM [Ray and vLLM](https://awslabs.github.io/ai-on-eks/docs/blueprints/inference/GPUs/vLLM-rayserve) to export their native metrics to Prometheus. You can also [connect Grafana to an AWS X-Ray data source](https://docs.aws.amazon.com/grafana/latest/userguide/x-ray-data-source.html) to visualize distributed traces and identify performance bottlenecks across your inference pipeline. This combination provides end-to-end visibility from GPU-level metrics through application-level request flows. @@ -56 +60 @@ Additionally, you can use open source frameworks like [Ray and vLLM](https://aws -For more information, refer to [Monitoring GPU workloads on Amazon EKS using AWS managed open-source services](https://aws.amazon.com/blogs/mt/monitoring-gpu-workloads-on-amazon-eks-using-aws-managed-open-source-services/) for a detailed implementation. +For step-by-step guidance on deploying this monitoring stack, refer to [Monitoring GPU workloads on Amazon EKS using AWS managed open-source services](https://aws.amazon.com/blogs/mt/monitoring-gpu-workloads-on-amazon-eks-using-aws-managed-open-source-services/). @@ -60 +64,22 @@ For more information, refer to [Monitoring GPU workloads on Amazon EKS using AWS -For core training metrics for AI/ML workloads on EKS, consider a combination of metrics that indicate the health and performance of your Amazon EKS cluster and the machine learning workloads running on it. Refer to [Introduction to observing machine learning workloads on Amazon EKS](https://aws.amazon.com/blogs/containers/part-1-introduction-to-observing-machine-learning-workloads-on-amazon-eks/) for a detailed implementation. Below, we break down the metrics we recommend monitoring. +Monitor core training metrics to track the health and performance of your Amazon EKS cluster and the machine learning workloads running on it. Training workloads have different monitoring requirements than inference workloads because they run for extended periods, consume resources differently, and require visibility into model convergence and data pipeline efficiency. The metrics below help you identify bottlenecks, optimize resource allocation, and ensure training jobs complete successfully. For step-by-step guidance on implementing this monitoring approach, refer to [Introduction to observing machine learning workloads on Amazon EKS](https://aws.amazon.com/blogs/containers/part-1-introduction-to-observing-machine-learning-workloads-on-amazon-eks/). + +**Resource Usage Metrics** + +Monitor resource usage metrics to validate that your resources are being properly consumed. These metrics help you identify bottlenecks and root cause performance issues. + + * **CPU, Memory, Network, GPU Power and GPU Temperature** \- Monitor these metrics to ensure allocated resources meet workload demands and identify optimization opportunities. Track metrics like gpu_memory_usage_bytes to identify memory consumption patterns and detect peak usage. Calculate percentiles such as the 95th percentile (P95) to understand the highest memory demands during training. This analysis helps you optimize models and infrastructure to avoid OOM errors and reduce costs. + + * **SM Occupancy, SM Activity, FPxx Activity** \- Monitor these metrics to understand how the underlying resource on the GPU is being used. Target 0.8 for SM Activity as a [rule of thumb](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html#metrics). + + * **Node and Pod Resource Utilization** \- Track resource usage at the node and pod level to identify resource contention and potential bottlenecks. Monitor whether nodes approach capacity limits, which can delay pod scheduling and slow training jobs. + + * **Resource Utilization Compared to Requests and Limits** — Compare actual resource usage against configured requests and limits to determine whether your cluster can handle current workloads and accommodate future ones. This comparison reveals whether you need to adjust resource allocations to avoid OOM errors or resource waste. + + * **Internal Metrics from ML Frameworks** \- Capture internal training and convergence metrics from ML frameworks such as TensorFlow and PyTorch. These metrics include loss curves, learning rate, batch processing time, and training step duration. Visualize these metrics using TensorBoard or similar tools to track model convergence and identify training inefficiencies. + + + + +**Model Performance Metrics** + +Monitor model performance metrics to validate that your training process produces models that meet accuracy and business requirements. These metrics help you determine when to stop training, compare model versions, and identify performance degradation. @@ -62 +87 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Resource Usage Metrics** : + * **Accuracy, Precision, Recall, and F1-score** — Track these metrics to understand how well your model performs on validation data. Calculate the F1-score on a validation set after each training epoch to assess whether the model is improving and when it reaches acceptable performance levels. @@ -64 +89 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **GPU and GPU Memory Usage** — Measure GPU and GPU memory utilization metrics to flag basic issues like idle resources or initial health checks in the cluster, but complement them with more precise saturation and usage metrics from the [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html). Augment GPU utilization with Power Usage to help highlight if the GPU is doing work. For a comprehensive approach, consider three levels of observability to ensure optimal GPU performance: 1) the topmost level checks if the GPU is active, 2) the middle level assesses Streaming Multiprocessor (SM) usage, 3) and the bottommost level evaluates workload efficiency on the SMs. Always view the system holistically, as bottlenecks may arise in CPU, memory, I/O, or network components rather than the GPU itself. + * **Business-Specific Metrics and KPIs** — Define and track metrics that directly measure the business value of your AI/ML initiatives. For a recommendation system, track metrics like click-through rate or conversion rate to ensure the model drives the intended business outcomes. @@ -66 +91 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Node and Pod Resource Utilization** — Tracking resource usage at the node and pod level helps you identify resource contention and potential bottlenecks. For example, check if any nodes are over-utilized, which could affect pod scheduling. + * **Performance over time** — Compare performance metrics across model versions and training runs to identify trends and detect degradation. Track whether newer model versions maintain or improve performance compared to baseline models. This historical comparison helps you decide whether to deploy new models or investigate training issues. @@ -68 +92,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Comparison of Resource Utilization with Requests and Limits** — This provides insight into whether your cluster can handle current workloads and accommodate future ones. For example, compare actual memory usage against limits to avoid out-of-memory errors. @@ -70 +93,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Internal Metrics from ML Frameworks** — Capture internal training and convergence metrics from ML frameworks (TensorFlow, PyTorch), such as loss curves, learning rate, batch processing time, and training step duration—often visualized with TensorBoard or similar. @@ -72,0 +96 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +**Data Quality and Drift Metrics** @@ -73,0 +98 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +Monitor data quality and drift metrics to ensure your training data remains consistent and representative. Data drift can cause model performance to degrade over time, while data quality issues can prevent models from converging or produce unreliable results. @@ -75 +100 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Model Performance Metrics** : + * **Statistical Properties of Input Data** — Track statistical properties such as mean, standard deviation, and distribution of input features over time to detect data drift or anomalies. Monitor whether feature distributions shift significantly from your baseline training data. For example, if the mean of a critical feature changes by more than two standard deviations, investigate whether your data pipeline has changed or whether the underlying data source has shifted. @@ -77 +102 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Accuracy, Precision, Recall, and F1-score** — These are vital for understanding the performance of your ML models. For example, after training, calculate the F1-score on a validation set to assess performance. + * **Data Drift Detection and Alerts** — Implement automated mechanisms to detect and alert on data quality issues before they impact training. Use statistical tests such as the Kolmogorov-Smirnov test or chi-squared test to compare current data distributions with your original training data. Set up alerts when tests detect significant drift so you can retrain models with updated data or investigate data pipeline issues. @@ -79 +103,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Business-Specific Metrics and KPIs** — Define and track metrics directly linked to the business value of your AI/ML initiatives. For example, in a recommendation system, track increased user engagement. @@ -81 +104,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Tracking these metrics over time** — This helps identify any degradation in model performance. For example, compare performance metrics across model versions to spot trends. @@ -83,0 +107 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +**Latency and Throughput Metrics** @@ -84,0 +109 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +Monitor latency and throughput metrics to identify bottlenecks in your training pipeline and optimize resource utilization. These metrics help you understand where time is spent during training and where to focus optimization efforts. @@ -86 +111 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Data Quality and Drift Metrics** : + * **End-to-End Latency of ML Training Pipelines** — Measure the total time for data to flow through your entire training pipeline, from data ingestion to model update. Track this metric across training runs to identify whether pipeline changes improve or degrade performance. High latency often indicates bottlenecks in data loading, preprocessing, or network communication between nodes. @@ -88 +113 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Statistical Properties of Input Data** — Monitor these over time to detect data drift or anomalies that could impact model performance. For example, track the mean of input features to detect shifts. + * **Training Throughput and Processing Rate** — Track the volume of data your training pipeline processes per unit of time to ensure efficient resource utilization. Monitor metrics such as samples processed per second or batches completed per minute. Low throughput relative to your hardware capacity suggests inefficiencies in data loading, preprocessing, or model computation that waste GPU cycles. @@ -90 +115 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Data Drift Detection and Alerts** — Implement mechanisms to automatically detect and alert on data quality issues. For example, use tests to compare current data with training data and alert on significant drift. + * **Checkpoint Save and Restore Latency** – Monitor the time required to save model checkpoints to storage (S3, EFS, FSx) and restore them to GPU or CPU memory when resuming jobs or recovering from failures. Slow checkpoint operations extend job recovery time and increase costs. Track checkpoint size, save duration, restore duration, and failure count to identify optimization opportunities such as compression or faster storage tiers. @@ -91,0 +117 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of + * **Data Loading and Preprocessing Time** \- Measure the time spent loading data from storage and applying preprocessing transformations. Compare this time against model computation time to determine whether your training is data-bound or compute-bound. If data loading consumes more than 20% of total training time, consider optimizing your data pipeline with caching, prefetching, or faster storage. @@ -95 +120,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Latency and Throughput Metrics** : @@ -97 +122 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **End-to-End Latency of ML Training Pipelines** — Monitor the time it takes for data to flow through the entire training pipeline. For example, measure total time from data ingestion to model update. +**Error Rates and Failures** @@ -99 +124 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Training Throughput and Processing Rate** — Track the volume of data processed during training to ensure efficiency. For example, monitor positive and negative samples processed per second. +Monitor error rates and failures throughout your training pipeline to maintain reliability and prevent wasted compute resources. Undetected errors can cause training jobs to fail silently, produce invalid models, or waste hours of GPU time before you notice problems. @@ -101 +126 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Checkpoint Restore Latency** – Monitor the time taken to load a saved model checkpoint from a storage location (S3, EFS, FSx etc) back to GPU/CPU memory when resuming a job, recovering from failure, or initializing. This metric directly impacts job recovery time, cold start performance, and overall efficiency of interference pipelines. In auto-scaling inference services, slow checkpoint loading can cause cold start delays and a degrading user experience. These related metrics are also commonly used to improve model checkpointing: checkpoint downtime latency, model deserialization time, checkpoint size, and checkpoint restore failure count. + * **Pipeline Error Monitoring** — Track errors across all stages of your ML pipeline, including data preprocessing, model training, and checkpoint operations. Log error types, frequencies, and affected components to quickly identify issues. Common errors include data format mismatches, out-of-memory failures during preprocessing, and checkpoint save failures due to storage limits. Set up alerts when error rates exceed baseline thresholds so you can investigate before errors cascade. @@ -103 +128 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Inference Request Duration** – Monitor the time it takes to complete an inference request. This is the time from initial request received to completed response from the model. + * **Recurring Error Analysis** — Identify and investigate patterns in recurring errors to prevent future failures and improve pipeline reliability. Analyze logs to find whether specific data samples, batch sizes, or training configurations consistently cause failures. For example, if certain input data types trigger preprocessing errors, add validation checks earlier in the pipeline or update your data cleaning logic. Track the mean time between failures (MTBF) to measure whether your pipeline reliability improves over time.“ @@ -105 +129,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Token Throughput** \- Monitor token processing time to gauge model performance and optimize scaling decisions. Slow processing can indicate inefficiencies or underutilized resources, impacting cost-effectiveness. Tracking metrics like tokens in per second and tokens out per second, alongside processing time, can help identify performance bottlenecks, slowdowns, and cost drivers. @@ -107 +130,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Identifying Performance Bottlenecks** — Use these metrics to pinpoint areas for optimization in the training process. For example, analyze time spent in data loading versus model computation. @@ -109,0 +133 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +**Kubernetes and EKS Specific Metrics** @@ -110,0 +135 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of +Monitor Kubernetes and EKS metrics to ensure your cluster infrastructure remains healthy and can support your training workloads. These metrics help you detect infrastructure issues before they cause training job failures or performance degradation. @@ -112 +137 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Error Rates and Failures** : + * **Kubernetes Cluster State Metrics** — Monitor the health and status of Kubernetes objects including pods, nodes, deployments, and services. Track pod status to identify pods stuck in pending, failed, or crash loop states. Monitor node conditions to detect issues like disk pressure, memory pressure, or network unavailability. Use kubectl or monitoring tools to check these metrics continuously and set up alerts when pods fail to start or nodes become unschedulable. @@ -114 +139 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Monitoring errors throughout the ML pipeline** — This includes data preprocessing, model training, and inference. For example, log errors in preprocessing to quickly identify issues. + * **Training Pipeline Execution Metrics** — Track successful and failed pipeline runs, job durations, step completion times, and orchestration errors. Monitor whether training jobs complete within expected time windows and whether failure rates increase over time. Track metrics such as job success rate, average job duration, and time to failure. These metrics help you identify whether infrastructure issues, configuration problems, or data quality issues cause training failures. @@ -116 +141 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Identifying and investigating recurring errors** — This helps maintain a high-quality model and ensure consistent performance. For example, analyze logs to find patterns like specific data causing failures. + * **AWS Service Metrics** — Track metrics for AWS services that support your EKS infrastructure and training workloads. Monitor S3 metrics such as request latency, error rates, and throughput to ensure data loading performance remains consistent. Track EBS volume metrics including IOPS, throughput, and queue length to detect storage bottlenecks. Monitor VPC flow logs and network metrics to identify connectivity issues between nodes or to external services. @@ -117,0 +143 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of + * **Kubernetes Control Plane Metrics** — Monitor the API server, scheduler, controller manager, and etcd database to detect performance issues or failures that affect cluster operations. Track API server request latency, request rate, and error rate to ensure the control plane responds quickly to scheduling requests. Monitor etcd database size, commit duration, and leader changes to detect stability issues. High API server latency or frequent etcd leader changes can delay pod scheduling and extend training job startup times. @@ -121 +146,0 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -**Kubernetes and EKS Specific Metrics** : @@ -123 +148 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Kubernetes Cluster State Metrics** — Monitor the health and status of various Kubernetes objects, including pods, nodes, and the control plane. For example, use tools like `kubectl` to check pod statuses. +**Application and Instance Logs** @@ -125 +150 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Success / Failed Pipeline Runs** — Track successful/failed pipeline runs, job durations, step completion times, and orchestration errors (e.g., using Kubeflow/Mlflow/Argo events). +Collect and analyze application and instance logs to diagnose issues that metrics alone cannot explain. Logs provide detailed context about errors, state changes, and system events that help you understand why training jobs fail or perform poorly. Correlating logs with metrics allows you to pinpoint root causes faster. @@ -127 +152 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **AWS Service Metrics** — Track metrics for other AWS services that support your EKS infrastructure and the applications running on it. For example, if using Amazon S3, monitor bucket size to track storage usage. + * **Application Logs** \- Collect application logs from your training jobs, data pipelines, and ML frameworks to identify bottlenecks and diagnose failures. These logs capture detailed information about job execution, including data loading errors, model initialization failures, checkpoint save errors, and framework-specific warnings. Correlate log timestamps with metric spikes to understand what caused performance degradation or failures. For example, if GPU utilization drops suddenly, check application logs for errors indicating data pipeline stalls or preprocessing failures. Use centralized logging tools like CloudWatch Logs or Fluent Bit to aggregate logs from all pods and make them searchable. @@ -129 +154 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of - * **Kubernetes Control Plane Metrics** — Monitor the API server, scheduler, controller manager, and etcd database for performance issues or failures. For example, track API server request latency for responsiveness. + * **Instance Logs** \- Collect instance-level logs such as system journal logs and dmesg output to detect hardware issues and kernel-level problems. These logs reveal issues like GPU driver errors, memory allocation failures, disk I/O errors, and network interface problems that may not appear in application logs. Correlate instance logs with application logs and metrics to determine whether training failures stem from hardware problems or application issues. For example, if a training job fails with an out-of-memory error, check dmesg logs for kernel OOM killer messages that indicate whether the system ran out of memory or whether the application exceeded its container limits. Set up alerts for critical hardware errors such as GPU XID errors or disk failures so you can replace failing instances before they cause widespread training disruptions. @@ -134 +159 @@ For core training metrics for AI/ML workloads on EKS, consider a combination of -In subsequent topics, we demonstrate gathering data for a few of the metrics mentioned above. We will provide examples with the two AWS recommended approaches: [AWS-native CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) and open-source [Amazon Managed Prometheus](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-getting-started.html) with [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html). You would choose one of these solutions based on your overall observability needs. See [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html) for the complete list of Container Insights metrics. +The following sections show how to collect the metrics described above using two AWS-recommended approaches: [CloudWatch Container Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-EKS.html) and Amazon Managed Prometheus [Amazon Managed Prometheus](https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-getting-started.html) with [Amazon Managed Grafana](https://docs.aws.amazon.com/grafana/latest/userguide/getting-started-with-AMG.html). Choose CloudWatch Container Insights if you prefer AWS-native tools with minimal setup and pre-built dashboards. Choose Amazon Managed Prometheus with Amazon Managed Grafana if you need customized dashboards, advanced visualization capabilities, or want to integrate with existing Prometheus-based monitoring infrastructure. For a complete list of available Container Insights metrics, see [Amazon EKS and Kubernetes Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-enhanced-EKS.html).