AWS emr documentation change
Summary
Added documentation about Spark dynamic allocation configuration to prevent excessive container churn
Security assessment
This change addresses performance and resource optimization by providing guidance on Spark executor configuration to avoid excessive container churn. While resource exhaustion could have security implications, this change is framed as a performance best practice without any security context or mention of vulnerabilities.
Diff
diff --git a/emr/latest/EMR-on-EKS-DevelopmentGuide/best-practices.md b/emr/latest/EMR-on-EKS-DevelopmentGuide/best-practices.md index 3a64e5297..418ed2718 100644 --- a//emr/latest/EMR-on-EKS-DevelopmentGuide/best-practices.md +++ b//emr/latest/EMR-on-EKS-DevelopmentGuide/best-practices.md @@ -62,0 +63,11 @@ For more information on security with Amazon EMR on EKS, see [Amazon EMR on EKS +By default, `spark.dynamicAllocation.preallocateExecutors` is enabled in Amazon EMR Spark. When `spark.dynamicAllocation.initialExecutors` and `spark.dynamicAllocation.minExecutors` are not set, Spark may request a large number of executors at startup based on estimated task counts, even for small workloads. To avoid excessive container churn, use one of the following approaches: + + * Set `spark.dynamicAllocation.initialExecutors` or `spark.dynamicAllocation.minExecutors` to a value appropriate for your workload size. + + * Set `spark.dynamicAllocation.preallocateExecutors.maxEstimatedTasks` to a lower value to limit the number of executors requested at startup. + + * Set `spark.dynamicAllocation.preallocateExecutors` to `false` to disable executor preallocation entirely. + + + +