AWS emr documentation change
Summary
Clarified node health monitoring logic, added disk space warning, expanded documentation about graceful replacement timeouts, and emphasized data backup requirements for HDFS
Security assessment
Changes focus on operational reliability and data loss prevention rather than addressing security vulnerabilities. While mentioning data backup recommendations improves resilience, there is no evidence of addressing authentication, authorization, or exploitation vulnerabilities.
Diff
diff --git a/emr/latest/ManagementGuide/emr-plan-node-replacement.md b/emr/latest/ManagementGuide/emr-plan-node-replacement.md index c325adae6..aefaa5154 100644 --- a//emr/latest/ManagementGuide/emr-plan-node-replacement.md +++ b//emr/latest/ManagementGuide/emr-plan-node-replacement.md @@ -9 +9 @@ Default node replacement and termination protection settingsConfiguring unhealth -Amazon EMR periodically uses the [NodeManager health checker service](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManager.html#Health_checker_service) in Apache Hadoop to monitor the statuses of core nodes in your Amazon EMR on Amazon EC2 clusters. If a node is not functionally optimally, the health checker reports that node to the Amazon EMR controller. The Amazon EMR controller adds the node to a denylist, preventing the node from receiving new YARN applications until the status of the node improves. One common reason of why a node might become unhealthy is because of overutilizing the disk. For more information about identifying unhealthy nodes and recovery, see [Resource errors](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-troubleshoot-error-resource.html). +Amazon EMR periodically uses the [NodeManager health checker service](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/NodeManager.html#Health_checker_service) in Apache Hadoop to monitor the statuses of core nodes in your Amazon EMR on Amazon EC2 clusters. If a node is not functioning optimally, the node is marked as unhealthy and the health checker reports that node to the Amazon EMR controller. The Amazon EMR controller adds the node to a deny list, preventing the node from receiving new YARN applications until the status of the node improves. @@ -11 +11 @@ Amazon EMR periodically uses the [NodeManager health checker service](https://ha -You can choose whether Amazon EMR should terminate unhealthy nodes or keep them in the cluster. If you turn off unhealthy node replacement, the unhealthy nodes stay in the denylist and continue to count towards cluster capacity. You can still connect to your Amazon EC2 core instance for configuration and recovery, so you can resize your cluster to add capacity. Note that Amazon EMR will replace unhealthy nodes even if [termination protection](https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_TerminationProtection.html) is on. +###### Note @@ -13 +13,9 @@ You can choose whether Amazon EMR should terminate unhealthy nodes or keep them -If unhealthy node replacement is on, Amazon EMR will terminate the unhealthy core node and provision a new instance based on the number of instances in the instance group or the target capacity for instance fleets. If multiple or all core nodes are unhealthy for more than 45 minutes, Amazon EMR will [gracefully replace the nodes](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-scaledown-behavior.html#emr-scaledown-terminate-task). +A common reason for a node to be unhealthy is that it is out of disk space. For more information about when a core node is almost out of disk space, the following **re:Post Knowledge Center** article is helpful: [Why is the core node in my Amazon EMR cluster running out of disk space?](https://repost.aws/knowledge-center/core-node-emr-cluster-disk-space) + +###### Note + +Hadoop does provide the ability to run customized node-health checks. This is explained in further detail in the Apache Hadoop documentation at [NodeManager](https://hadoop.apache.org/docs/r3.3.2/hadoop-yarn/hadoop-yarn-site/NodeManager.html). + +You can choose whether Amazon EMR should terminate unhealthy nodes or keep them in the cluster. If you turn off unhealthy-node replacement, they stay in the deny list and continue to count toward cluster capacity. You can still connect to your Amazon EC2 core instance for configuration and recovery, so you can resize your cluster if you want to add capacity. For more information about how node replacement and termination work, see [Using termination protection](https://docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_TerminationProtection.html). + +If unhealthy node replacement is turned on, Amazon EMR terminates an unhealthy core node and provisions a new instance, based on the number of instances in the instance group, or based on the target capacity for instance fleets. If any nodes are unhealthy for more than 45 minutes, Amazon EMR will [gracefully replace the nodes](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-scaledown-behavior.html#emr-scaledown-terminate-task). If graceful decommissioning for a node doesn't complete within one hour, the node is forcefully terminated, unless terminating it brings the cluster below replication factor or HDFS capacity constraints. @@ -17 +25,5 @@ If unhealthy node replacement is on, Amazon EMR will terminate the unhealthy cor -To avoid the possibility of permanently losing HDFS data as Amazon EMR gracefully replaces an unhealthy core instance, we recommend that you always back up your data. +Note that the time it takes before a node is gracefully decommissioned or terminated can be subject to change. + +While unhealthy node replacement significantly mitigates the possibility for data loss, it doesn't eliminate the risk entirely. HDFS data can be permanently lost during the graceful replacement of an unhealthy core instance. We recommend that you always back up your data. + +For more information about identifying unhealthy nodes and recovery, see [Resource errors](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-troubleshoot-error-resource.html). Additionally, for more best practices you can follow in order to maintain the health of a cluster, see the following documentation for the resource error [Amazon EMR cluster terminates with NO_SLAVE_LEFT and core nodes FAILED_BY_MASTER](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-cluster-NO_SLAVE_LEFT-FAILED_BY_MASTER.html).