AWS glue documentation change
Summary
Clarified Spark shuffle storage implementation using Cloud Shuffle Storage Plugin
Security assessment
Added explicit reference to Cloud Shuffle Storage Plugin for better feature clarity without security implications
Diff
diff --git a/glue/latest/dg/monitor-spark-shuffle-manager.md b/glue/latest/dg/monitor-spark-shuffle-manager.md index e5f7630ba..78337dd85 100644 --- a//glue/latest/dg/monitor-spark-shuffle-manager.md +++ b//glue/latest/dg/monitor-spark-shuffle-manager.md @@ -19 +19 @@ AWS Glue Spark shuffle plugin with Amazon S3 is only supported for AWS Glue ETL -With AWS Glue, you can now use Amazon S3 to store Spark shuffle data. Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. This solution disaggregates compute and storage for your Spark jobs, and gives complete elasticity and low-cost shuffle storage, allowing you to run your most shuffle-intensive workloads reliably. +With AWS Glue, you can now use Amazon S3 to store Spark shuffle data using the Cloud Shuffle Storage Plugin. Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance. This solution disaggregates compute and storage for your Spark jobs, and gives complete elasticity and low-cost shuffle storage, allowing you to run your most shuffle-intensive workloads reliably. @@ -23 +23 @@ With AWS Glue, you can now use Amazon S3 to store Spark shuffle data. Amazon S3 -We are introducing a new Cloud Shuffle Storage Plugin for Apache Spark to use Amazon S3. You can turn on Amazon S3 shuffling to run your AWS Glue jobs reliably without failures if they are known to be bound by the local disk capacity for large shuffle operations. In some cases, shuffling to Amazon S3 is marginally slower than local disk (or EBS) if you have a large number of small partitions or shuffle files written out to Amazon S3. +You can turn on Amazon S3 shuffling to run your AWS Glue jobs reliably without failures if they are known to be bound by the local disk capacity for large shuffle operations. In some cases, shuffling to Amazon S3 is marginally slower than local disk (or EBS) if you have a large number of small partitions or shuffle files written out to Amazon S3.