AWS parallelcluster documentation change
Summary
Added structured sections for root causing, preventing, and recovering from cluster update failures, including specific commands to detect broken nodes and restart services.
Security assessment
The changes are operational troubleshooting guides for cluster updates. While they mention preventing update failures by terminating broken nodes, there is no evidence of addressing a specific security vulnerability. The changes focus on reliability and recovery procedures, not security features or vulnerabilities.
Diff
diff --git a/parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md b/parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md index 20a80231a..f572d463c 100644 --- a//parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md +++ b//parallelcluster/latest/ug/troubleshooting-fc-v3-update-cluster.md @@ -17 +17,5 @@ Check the `~/.parallelcluster/pcluster-cli.log` in your local file system for fa -If the cluster stack update rolled back, check the `/var/log/chef-client.log` file for error details. +### Root causing + +To identify the root cause of the failure, the starting point is to look at cluster stack events and `/var/log/chef-client.log` in the head node. + +A possible cause is that at least one cluster node did not apply the update. You can retrieve the list of nodes that failed to update in `/var/log/chef-client.log` in the head node by looking for `Check cluster readiness` in the log. @@ -21 +25,16 @@ Check to see if your issue is mentioned in [GitHub Known Issues](https://github. -If the rollback failed according to logs `/var/log/chef-client.log`, it may be that `clustermgtd` was stopped to prevent the amplification of failures. In this case, you need to manually restart it by executing the following command on the Head Node: +### Preventing + +A cluster update can fail if at least one node in the cluster did not successfully apply the update. To reduce the risk of cluster update failure, we recommend terminating broken nodes before initiating the update. An example of nodes that could be broken are compute nodes stuck in `COMPLETING` state for longer than the expected epilog duration. To detect those nodes, you can run the following command, adapting the `threshold` value to your needs (the value must be greater than the maximum duration expected for your epilogs). + + + $ scontrol show nodes --json | jq -r --argjson threshold 60 ' + .nodes[] | select(.state | index("COMPLETING")) | + select((now - .last_busy.number) > $threshold) | + .name + ' + +### Recovering + +If the update failed, the rollback is the mechanism expected to recover the state of the cluster. + +If the rollback failed, the cluster state is not deterministic. In this case, it may be that `clustermgtd` was stopped to prevent the amplification of failures. We recommend starting it by running the following command on the head node. Adapt the Python version to the one shipped with your AWS ParallelCluster version: