AWS wellarchitected documentation change
Summary
Added detailed guidance about monitoring HPC job performance, implementing performance testing procedures, and integrating logging/dashboards. Reorganized content around new HPCOPS06 best practice framework.
Security assessment
Changes focus on performance monitoring and operational excellence rather than addressing security vulnerabilities. While improved monitoring could help detect anomalies, there's no explicit mention of security controls, vulnerabilities, or security-related use cases. The primary focus is on cost optimization and performance reliability.
Diff
diff --git a/wellarchitected/latest/high-performance-computing-lens/operate.md b/wellarchitected/latest/high-performance-computing-lens/operate.md index 294632829..02651abdc 100644 --- a//wellarchitected/latest/high-performance-computing-lens/operate.md +++ b//wellarchitected/latest/high-performance-computing-lens/operate.md @@ -3 +3,3 @@ -[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](welcome.html) +[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](high-performance-computing-lens.html) + +HPCOPS06-BP01 Test and observe job-level performance for every change @@ -7 +9,2 @@ -Operations must be standardized and managed routinely. Focus on automation, small frequent changes, regular quality assurance testing, and defined mechanisms to track, audit, roll back, and review changes. Changes should not be large and infrequent, should not require scheduled downtime, and should not require manual execution. A wide range of logs and metrics based on key operational indicators for a workload must be collected and reviewed to ensure continuous operations. +HPCOPS06: How do you monitor your workloads to verify they are operating as expected? +--- @@ -9 +12 @@ Operations must be standardized and managed routinely. Focus on automation, smal -AWS provides the opportunity to use additional tools for handling HPC operations. These tools can vary from monitoring assistance to automating deployments. For example, you can have Auto Scaling restart failed instances, use CloudWatch to monitor your cluster’s load metrics, configure notifications for when jobs finish, or use a managed service (such as AWS Batch) to implement retry rules for failed jobs. Cloud-native tools can greatly improve your application deployment and change management. +In HPC environments, job-level performance is often one of the most important characteristics not just for end-user experience but also for cost optimization. This performance can be affected by a number of different factors and updates or changes can cause unexpected performance impacts. Your environment is also likely to change frequently with small updates such as for packages and drivers, and any impact of these may go unaccounted for if we only test performance for major upgrades. In such cases, it may be worthwhile creating a procedure that all changes have to go through before being implemented, and adding a step to this procedure which tests if the change impacts your runtime performance. @@ -11 +14 @@ AWS provides the opportunity to use additional tools for handling HPC operations -Release management processes, whether manual or automated, must be based on small incremental changes and tracked versions. You must be able to revert releases that introduce issues without causing operational impact. Use continuous integration and continuous deployment tools such as AWS CodePipeline and AWS CodeDeploy to automate change deployment. Track source code changes with version control tools and infrastructure configurations with automation tools, such as AWS CloudFormation templates. +## HPCOPS06-BP01 Test and observe job-level performance for every change @@ -13,2 +16 @@ Release management processes, whether manual or automated, must be based on smal -HPCOPS 3: How are you evolving your workload while minimizing the impact of change? ---- +Before moving users to a new cluster environment, you should run a set of representative HPC job benchmarks to confirm that your system is performing as expected. To verify that this performance is maintained, consider periodically rerunning these benchmarks or a subset so that any unexpected changes can be localized and investigated early. @@ -16,2 +18,11 @@ HPCOPS 3: How are you evolving your workload while minimizing the impact of chan -HPCOPS 4: How do you monitor your workload to ensure that it is operating as expected? ---- +As the projects of your users evolve, the requirements and usage of your HPC environment will also change from the initial set of jobs on which your representative set of testcases were built. To verify that the performance tests are relevant, you could periodically update your testcases, or you can consider alternative methods such as monitoring the jobs that your users are running. For example, by monitoring job logs or percent usage by department or user, you may proactively detect anomalies. You can then investigate whether these anomalies were caused by a known change in usage patterns, or an unexplained performance regression. Set alerts and automated responses where appropriate. + +Performance regressions can go undetected as they may not throw any errors, but can result in longer running jobs and increased cost per job. Consider adding operational mechanisms to track metrics of your jobs and building them into a cohesive dashboard. You can use these collected metrics to tune your environment based on real usage, such as rightsizing the tier of throughput and capacity of your file systems, or adding new compute options similar to hardware configurations that are currently oversubscribed. + +### Implementation guidance + +Log job-level statistics, track anomalies and integrate your environment logging into a dashboard. + +There are a number of options for tracking the operational performance of your HPC environment which vary in the level of granularity the offer and operational overhead required to run them. Most HPC schedulers have their own tools to track job level metrics, and these can be the easiest place to start as they natively integrate with the scheduler. + +If using AWS ParallelCluster with Slurm, leverage [Slurm accounting with AWS ParallelCluster](https://docs.aws.amazon.com/parallelcluster/latest/ug/slurm-accounting-v3.html) to log job-level statistics in an external database. You can then add a method to visualize these metrics so you can easily gain a view across your environment. The [ParallelCluster Monitoring dashboard](https://github.com/aws-samples/parallelcluster-monitoring-dashboard) repository is an example of how you can construct a dashboard to track job data. If using AWS Batch, a similar tool is the [AWS Batch Runtime Monitoring Dashboards Solution](https://github.com/aws-samples/aws-batch-runtime-monitoring). @@ -19 +30 @@ HPCOPS 4: How do you monitor your workload to ensure that it is operating as exp -Using the cloud for HPC introduces new operational considerations. While on-premises clusters are fixed in size, cloud clusters can scale to meet demand. Cloud-native architectures for HPC also operate differently than on-premises architectures. For example, they use different mechanisms for job submission and provisioning On-Demand Instance resources as jobs arrive. You must adopt operational procedures that accommodate the elasticity of the cloud and the dynamic nature of cloud-native architectures. +Higher level alternative or complementary tracking methods such as tagging cloud resources by project and using them to drill down into cost reports using AWS cost allocation tags to detect anomalies can offer a similar effect with lower operational overhead but reduced granularity. For more information, see [Organizing and tracking costs using AWS cost allocation tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html). Many tools such as [AWS ParallelCluster resources and tagging](https://docs.aws.amazon.com/parallelcluster/latest/ug/resources-tags-v3.html) and AWS Batch resource tagging: [Tag your AWS Batch](https://docs.aws.amazon.com/batch/latest/userguide/using-tags.html) resources integrate with this mechanism natively to simplify automated tagging.