AWS omics documentation change
Summary
Added new section 'Guidance for unresponsive runs' with best practices for preventing, catching, and troubleshooting stuck workflows
Security assessment
The changes focus on operational reliability and cost management rather than addressing security vulnerabilities. While they mention resource allocation and process management, there's no evidence of patching vulnerabilities or addressing security incidents.
Diff
diff --git a/omics/latest/dev/workflows-run-errors.md b/omics/latest/dev/workflows-run-errors.md index 0999bf5fd..9facc00c9 100644 --- a/omics/latest/dev/workflows-run-errors.md +++ b/omics/latest/dev/workflows-run-errors.md @@ -4,0 +5,2 @@ +Guidance for unresponsive runs + @@ -35,0 +38,53 @@ Failure reason | Error description +## Guidance for unresponsive runs + +When developing new workflows, runs or specific tasks could become "stuck" or "hang" if there are issues with your code, and tasks fail to exit processes properly. This can be challenging to troubleshoot and catch, as it is normal for tasks to run for extended periods. To prevent and identify unresponsive runs, follow the suggested best practices in the following sections. + +**Best practices for preventing unresponsive runs** + + * Ensure you are closing all the files opened in your task code. Opening too many files can ocassionally lead to threading issues within the workflow engines. + + * Background processes created by a workflow task should exit when the task exits. However, if a background process does not exit cleanly, you must explicitly shut down that process in your task code. + + * Ensure your processes do not loop without exiting. This can cause an unresponsive run, and requires a change to your workflow definition code to resolve. + + * Provide appropriate memory and CPU allocation to your tasks. Analyze the [CloudWatch logs](https://docs.aws.amazon.com/omics/latest/dev/monitoring-cloudwatch-logs.html) or use the [Run Analyzer](./workflows-run-optimize.html#workflows-run-analyzer) on successfully completed runs of your workflow to verify you have optimal compute allocation. Use the Run Analyzer `headroom` parameter to include additional headroom, ensuring processes have sufficient resources to complete. You should have at least 5% headroom of allocated memory for background operating system processes. + + * Ensure you are using the correct file system size for your runs. For unresponsive runs that are using static run storage, consider increasing the static run storage allocation to enable higher IO throughput and storage capacity on the file system. Analyze the run manifest to see the maximum file system storage, use the Run Analyzer to determine if the file system allocation needs to be increased. + + + + +**Best practices for catching unresponsive runs** + + * When developing new workflows, use a run group with the max run time limit set to catch runaway code. For instance, if a run should take 1 hour to complete, place it in a run group that times out after 2 or 3 hours (or a different time period based on your use case) to catch run-away jobs. Also, apply a buffer to account for variance in processing times. + + * Set up a series of run groups with different maximum runtime limits. For instance, you could assign short runs to a run group that terminates the runs after a few hours, and a long runs group that terminates runs after a few days, based on your expected workflow duration. + + * HealthOmics has a default maximum run duration service limit of 604,800 Seconds, or 7 days, which is adjustable through a request in the quotas tool. Only request a service limit increase of this quota if you have runs that approach a week in duration. If you have a mix of short and long runs and are not using run groups, consider putting the long-running runs in a separate account with a higher maximum run duration service limit. + + * Inspect the [CloudWatch logs](https://docs.aws.amazon.com/omics/latest/dev/monitoring-cloudwatch-logs.html) for tasks that you suspect could be unresponsive. If a task normally outputs regular log statements and has not done so for an extended period, the task is likely stuck or frozen. + + + + +**What to do if you encounter an unresponsive run** + + * Cancel the run to avoid incurring additional costs. + + * Inspect the [task logs](https://docs.aws.amazon.com/omics/latest/dev/monitoring-cloudwatch-logs.html#cloudwatch-logs) to check if any processes failed to exit correctly. + + * Inspect the [engine logs](https://docs.aws.amazon.com/omics/latest/dev/monitoring-cloudwatch-logs.html#cloudwatch-logs) to identify any abnormal engine behaviors. + + * Compare the task and engine logs from the unresponsive run to those of identical, successfully completed runs. This can help identify any differences that may have caused the unresponsive behavior. + + * If you are unable to determine the root cause, raise a [support case](https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case) and include the following: + + * ARN of the stuck run and ARN of an identical run that completed successfully. + + * Engine logs (available once the run has been cancelled or fails) + + * Task logs for the unresponsive task. We don't require task logs for all tasks in the workflow to troubleshoot. + + + +