AWS prescriptive-guidance documentation change
Summary
Updated image paths, converted section headings to markdown level 4 (####), reformatted note about Spark collect() best practices
Security assessment
Changes involve documentation formatting, image path updates, and minor editorial improvements. The note about avoiding Spark collect() addresses performance/reliability (OOM errors), not security vulnerabilities. No security-specific content was added or modified.
Diff
diff --git a/prescriptive-guidance/latest/tuning-aws-glue-for-apache-spark/optimize-shuffles.md b/prescriptive-guidance/latest/tuning-aws-glue-for-apache-spark/optimize-shuffles.md index 5d0a9d84f..1aa3a3696 100644 --- a//prescriptive-guidance/latest/tuning-aws-glue-for-apache-spark/optimize-shuffles.md +++ b//prescriptive-guidance/latest/tuning-aws-glue-for-apache-spark/optimize-shuffles.md @@ -38 +38 @@ If the **Shuffle Bytes Written** value is high compared with **Shuffle Bytes Rea - + @@ -48 +48 @@ The **Shuffle Spill (Disk)** column shows a large amount of data spill memory to - + @@ -65 +65 @@ After you confirm that your join operations are essential to your business requi -_Use pushdown before join_ +#### Use pushdown before join @@ -88 +88 @@ Filter out unnecessary rows and columns in the DataFrame before performing a joi -_Use DataFrame Join_ +#### Use DataFrame Join @@ -92 +92 @@ Try using a [Spark high-level API](https://archive.apache.org/dist/spark/docs/3. -_Shuffle and broadcast hash joins and hints_ +#### Shuffle and broadcast hash joins and hints @@ -98 +98 @@ The following diagram shows the high-level structure and steps of a broadcast ha - + @@ -157 +157 @@ In AWS Glue 3.0 and or later, you can use other join hints for shuffle to tune y -_Use bucketing_ +#### Use bucketing @@ -161 +161 @@ _The sort-merge join_ requires two phases, shuffle and sort, and then merge. The - + @@ -177 +177 @@ You can create a bucketed table by using the following code. -_Repartition DataFrames on join keys before the join_ +#### Repartition DataFrames on join keys before the join @@ -197 +197 @@ To identify data skew, assess the following metrics in the Spark UI: - + @@ -201 +201 @@ To identify data skew, assess the following metrics in the Spark UI: - + @@ -263,3 +263,2 @@ If you don't need `collect()` or other actions in your commercial environment, c -###### Note - -Avoid using Spark `collect()` in commercial environments as much as possible. The `collect()` action returns all the results of a calculation in the Spark executor to the Spark driver, which might cause the Spark driver to return an OOM error. To avoid an OOM error, Spark sets `spark.driver.maxResultSize = 1GB` by default, which limits the maximum data size returned to the Spark driver to 1 GB. +Note: Avoid using Spark `collect()` in commercial environments as much as possible. The `collect()` action returns all the results of a calculation in the Spark executor to the Spark driver, which might cause the Spark driver to return an OOM error. To avoid an OOM error, Spark sets `spark.driver.maxResultSize = 1GB` by default, which limits the maximum data size returned to the Spark driver to 1 GB. +---