AWS prescriptive-guidance documentation change
Summary
Updated EMR version references, simplified Iceberg catalog configuration syntax, and removed redundant parameters
Security assessment
The changes update technical documentation for newer EMR versions and configuration best practices. While the removal of explicit S3FileIO configuration could impact data handling, there's no explicit evidence of addressing security vulnerabilities or adding security controls in the diff.
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-emr.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-emr.md index 93a01a07b..9f92d4855 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-emr.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-emr.md @@ -19 +19 @@ Amazon EMR supports multiple deployment options: Amazon EMR on Amazon EC2, Amazo -Amazon EMR version 6.5.0 and later versions support Apache Iceberg natively. For a list of supported Iceberg versions for each Amazon EMR release, see [Iceberg release history](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Iceberg-release-history.html) in the Amazon EMR documentation. Also review [considerations and limitations for using Iceberg on Amazon EMR](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-considerations.html) to see which Iceberg features are supported in Amazon EMR on different frameworks. +Amazon EMR version 6.5.0 and later versions support Apache Iceberg natively. For a list of supported Iceberg versions for each Amazon EMR release, see [Iceberg release history](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/Iceberg-release-history.html) in the Amazon EMR documentation. Also review the sections under [Use a cluster with Iceberg](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-iceberg-use-cluster.html) to see which Iceberg features are supported in Amazon EMR on different frameworks. @@ -21 +21 @@ Amazon EMR version 6.5.0 and later versions support Apache Iceberg natively. For -We recommend that you use the latest Amazon EMR version to benefit from the latest supported Iceberg version. The code examples and configurations in this section assume that you're using Amazon EMR release **emr-6.9.0**. +We recommend that you use the latest Amazon EMR version to benefit from the latest supported Iceberg version. The code examples and configurations in this section assume that you're using Amazon EMR release **emr-7.8.0**. @@ -60,2 +60 @@ To use Iceberg in EMR Studio, follow these steps: - "spark.sql.catalog.<catalog_name>.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", - "spark.sql.catalog.<catalog_name>.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", + "spark.sql.catalog.<catalog_name>.type": "glue", @@ -68 +68 @@ where: - * `<catalog_name>` is your Iceberg Spark session catalog name. Replace it with the name of your catalog, and remember to change the references throughout all configurations that are associated with this catalog. In your code, you should then refer to your Iceberg tables with the fully qualified table name, including the Spark session catalog name, as follows: + * `<catalog_name>` is your Iceberg Spark session catalog name. Replace it with a name of your choice, and remember to change the references throughout all configurations that are associated with this catalog. In your code, you can refer to your Iceberg tables by using the fully qualified table name, including the Spark session catalog name, as follows: @@ -72 +72 @@ where: - * `<catalog_name>.warehouse` points to the Amazon S3 path where you want to store your data and metadata. +Alternatively, you can change the default catalog to the Iceberg catalog that you defined by setting `spark.sql.defaultCatalog` to your catalog name. This second approach enables you to refer to tables without the catalog prefix, which can simplify your queries. @@ -74 +74 @@ where: - * To make the catalog an AWS Glue Data Catalog, set `<catalog_name>.catalog-impl `to `org.apache.iceberg.aws.glue.GlueCatalog`. This key is required to point to an implementation class for any custom catalog implementation. The [General best practices](./best-practices-general.html) section later in this guide describes the different Iceberg-supported catalogs. + * `<catalog_name>.warehouse` points to the Amazon S3 path where you want to store your data and metadata. @@ -76 +76 @@ where: - * Use `org.apache.iceberg.aws.s3.S3FileIO` as the `<catalog_name>.io-impl` in order to take advantage of Amazon S3 multipart upload for high parallelism. + * To make the catalog an AWS Glue Data Catalog, set `spark.sql.catalog.<catalog_name>.type` to `glue`. This key is required to point to an implementation class for any custom catalog implementation. The [General best practices](./best-practices-general.html) section later in this guide describes the different Iceberg-supported catalogs. @@ -120 +120 @@ You can use these steps to submit the Iceberg Spark job: - **"spark.sql.catalog. <catalog_name>.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog",** + **"spark.sql.catalog. <catalog_name>.type=glue",** @@ -123,3 +123 @@ You can use these steps to submit the Iceberg Spark job: - "--conf", - **"spark.sql.catalog. <catalog_name>.io-impl=org.apache.iceberg.aws.s3.S3FileIO", - "s3://YOUR-BUCKET-NAME/code/iceberg-job.py"** + "s3://YOUR-BUCKET-NAME/code/iceberg-job.py" @@ -146,0 +145 @@ To submit an Iceberg Spark job to Amazon EMR Serverless by using the AWS CLI: + "name": "iceberg-test-job", @@ -150,2 +149 @@ To submit an Iceberg Spark job to Amazon EMR Serverless by using the AWS CLI: - "entryPointArguments": [], - "sparkSubmitParameters": "--jars /usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar" + "entryPointArguments": [] @@ -158,6 +156,5 @@ To submit an Iceberg Spark job to Amazon EMR Serverless by using the AWS CLI: - **"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",** - **"spark.sql.catalog. <catalog_name>": "org.apache.iceberg.spark.SparkCatalog",** - **"spark.sql.catalog. <catalog_name>.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",** - **"spark.sql.catalog. <catalog_name>.warehouse": "s3://YOUR-BUCKET-NAME/YOUR-FOLDER-NAME/",** - **"spark.sql.catalog. <catalog_name>.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",** - **"spark.jars":"/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar",** + "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + "spark.sql.catalog.<catalog_name>": "org.apache.iceberg.spark.SparkCatalog", + "spark.sql.catalog.<catalog_name>.type": "glue", + "spark.sql.catalog.<catalog_name>.warehouse": "s3://YOUR-BUCKET-NAME/YOUR-FOLDER-NAME/", + "spark.jars":"/usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar", @@ -208 +205 @@ To submit an Iceberg Spark job to Amazon EMR on Amazon EKS by using the AWS CLI: - "sparkSubmitParameters": **"--jars local:///usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar"** + "sparkSubmitParameters": "--jars local:///usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar" @@ -215,5 +212,4 @@ To submit an Iceberg Spark job to Amazon EMR on Amazon EKS by using the AWS CLI: - **"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",** - **"spark.sql.catalog. <catalog_name>": "org.apache.iceberg.spark.SparkCatalog",** - **"spark.sql.catalog. <catalog_name>.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",** - **"spark.sql.catalog. <catalog_name>.warehouse": "s3://YOUR-BUCKET-NAME/YOUR-FOLDER-NAME/",** - **"spark.sql.catalog. <catalog_name>.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",** + "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + "spark.sql.catalog.<catalog_name>": "org.apache.iceberg.spark.SparkCatalog", + "spark.sql.catalog.<catalog_name>.type": "glue", + "spark.sql.catalog.<catalog_name>.warehouse": "s3://YOUR-BUCKET-NAME/YOUR-FOLDER-NAME/",