AWS prescriptive-guidance documentation change
Summary
Updated documentation for inserting data into partitioned Iceberg tables, removing explicit sorting steps and adding note about Iceberg 1.5.0's default hash distribution mode
Security assessment
Changes focus on query syntax updates and documentation of default write distribution behavior, with no security-related content or vulnerabilities addressed
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-spark.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-spark.md index 7da5532ac..52470adce 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-spark.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-spark.md @@ -65 +65 @@ Here's an example of creating a partitioned Iceberg table with Spark SQL: -To insert data into a partitioned Iceberg table with Spark SQL, you perform a global sort and then write the data: +To insert data into a partitioned Iceberg table with Spark SQL, use a standard `INSERT INTO` statement: @@ -72 +71,0 @@ To insert data into a partitioned Iceberg table with Spark SQL, you perform a gl - ORDER BY c_birth_country @@ -74,0 +74,4 @@ To insert data into a partitioned Iceberg table with Spark SQL, you perform a gl +###### Note + +Starting with Iceberg 1.5.0, `hash` write distribution mode is the default when you insert data into partitioned tables. For more information, see [Writing Distribution Modes](https://iceberg.apache.org/docs/latest/spark-writes/#writing-distribution-modes) in the Iceberg documentation. + @@ -98 +101 @@ To insert data into an existing unpartitioned Iceberg table by using the `DataFr -To create and populate a partitioned Iceberg table by using the `DataFrameWriterV2` API, you can use a local sort to ingest data: +To create and populate a partitioned Iceberg table by using the `DataFrameWriterV2` API: @@ -101,2 +104 @@ To create and populate a partitioned Iceberg table by using the `DataFrameWriter - input_data.sortWithinPartitions("c_birth_country") \ - .writeTo(f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_withpartitions") \ + input_data.writeTo(f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_withpartitions") \ @@ -107 +109 @@ To create and populate a partitioned Iceberg table by using the `DataFrameWriter -To insert data into a partitioned Iceberg table by using the `DataFrameWriterV2` API, you can use a global sort to ingest data: +To insert data into a partitioned Iceberg table by using the `DataFrameWriterV2` API: @@ -110,2 +112 @@ To insert data into a partitioned Iceberg table by using the `DataFrameWriterV2` - input_data.orderBy("c_birth_country") \ - .writeTo(f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_withpartitions") \ + input_data.writeTo(f"{CATALOG_NAME}.{DB_NAME}.{TABLE_NAME}_withpartitions") \