AWS parallelcluster documentation change
Summary
Reorganized troubleshooting content and added detailed procedures for cluster update failures. Added new sections for handling UPDATE_ROLLBACK_FAILED status and stacks stuck in UPDATE_IN_PROGRESS/UPDATE_ROLLBACK_IN_PROGRESS states, including specific AWS CLI commands for recovery.
Security assessment
The changes focus on operational troubleshooting of cluster updates and rollbacks without addressing security vulnerabilities, exploits, or security features. No security-related terminology or mitigation of security risks is present.
Diff
diff --git a/parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md b/parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md index 51f4b1821..6890e4c23 100644 --- a//parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md +++ b//parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md @@ -7 +7 @@ -pcluster update-cluster command fails to run locallySeeing clusterStatus is UPDATE_FAILED with pcluster describe-cluster commandThe cluster update timed out +pcluster update-cluster command fails to run locallyThe cluster update timed outclusterStatus is UPDATE_FAILEDThe cluster stack appears stuck in UPDATE_IN_PROGRESS or UPDATE_ROLLBACK_IN_PROGRESS @@ -17 +17,5 @@ Check the `~/.parallelcluster/pcluster-cli.log` in your local file system for fa -## Seeing `clusterStatus` is `UPDATE_FAILED` with `pcluster describe-cluster` command +## The cluster update timed out + +This could be an issue related to `cfn-hup` not running. If the `cfn-hup` demon is terminated by an external cause, it's not restarted automatically. If `cfn-hup` isn't running, during a cluster update, the CloudFormation stack starts the update process as expected, but the update procedure isn't activated on the head node and the stack deployment eventually times out. For more information, see [Troubleshooting a cluster update timeout when cfn-hup isn't running](./troubleshooting-v3-cluster-update-timeout.html) to troubleshoot and recover from the issue. + +## `clusterStatus` is `UPDATE_FAILED` @@ -47 +51 @@ If the rollback failed, the cluster state is not deterministic. In this case, it -## The cluster update timed out +### `clusterStatus` is `UPDATE_FAILED` and `cloudFormationStackStatus` is `UPDATE_ROLLBACK_FAILED` @@ -49 +53,31 @@ If the rollback failed, the cluster state is not deterministic. In this case, it -This could be an issue related to `cfn-hup` not running. If the `cfn-hup` demon is terminated by an external cause, it's not restarted automatically. If `cfn-hup` isn't running, during a cluster update, the CloudFormation stack starts the update process as expected, but the update procedure isn't activated on the head node and the stack deployment eventually times out. For more information, see [Troubleshooting a cluster update timeout when cfn-hup isn't running](./troubleshooting-v3-cluster-update-timeout.html) to troubleshoot and recover from the issue. +When `pcluster describe-cluster` reports that `clusterStatus` is `UPDATE_FAILED` and `cloudFormationStackStatus` is `UPDATE_ROLLBACK_FAILED`, both the cluster update and the subsequent rollback failed. In this state the cluster stack cannot accept any further update and requires manual intervention to be unblocked. + +To unblock the cluster stack, complete the following steps: + + 1. Fix the root cause of the failure. + + 2. Force the continuation of the rollback. + + $ aws cloudformation continue-update-rollback --region REGION --stack-name CLUSTER_STACK_NAME + + 3. Wait for the stack to reach the `UPDATE_ROLLBACK_COMPLETE` status. + + 4. Retry the original update with the `pcluster update-cluster` command. + + + + +## The cluster stack appears stuck in `UPDATE_IN_PROGRESS` or `UPDATE_ROLLBACK_IN_PROGRESS` + +The cluster stack could be stuck in `UPDATE_IN_PROGRESS` or `UPDATE_ROLLBACK_IN_PROGRESS` for 1 hour at most, if the login nodes Auto Scaling Group is failing to stabilize due to bootstrap failures in the login nodes. + +To verify whether you are in this situation due to the login nodes Auto Scaling Group, you need to do the following checks: In the main stack, the only resource that is in `UPDATE_IN_PROGRESS` is the login nodes nested stack. In the login nodes nested stack, the only resource that is stuck in `UPDATE_IN_PROGRESS` is the login nodes Auto Scaling Group. + +If you are in this scenario, you can cancel the update so that you do not need to wait 1 hour for the update to complete. + + + $ aws cloudformation cancel-update-stack --region REGION --stack-name CLUSTER_STACK_NAME + +Canceling the update triggers the rollback. You cannot reduce the 1 hour timeout on the rollback, so in the worst case scenario you need to wait for the rollback to reach its final state. + +If the rollback succeeds, you can immediately retry your original cluster update once you have fixed the root cause of the failure. Otherwise, see clusterStatus is UPDATE_FAILED and cloudFormationStackStatus is UPDATE_ROLLBACK_FAILED.