AWS emr documentation change
Summary
Restructured Iceberg configuration example and added Hudi/Delta Lake configuration samples with Lake Formation integration
Security assessment
Added documentation for security-sensitive configurations (AWS Glue endpoints, Lake Formation access). While security-related, this is routine feature documentation rather than addressing a specific vulnerability.
Diff
diff --git a/emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable.md b/emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable.md index 808d44e95..b5788089f 100644 --- a//emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable.md +++ b//emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable.md @@ -149,8 +149 @@ After you set up the Lake Formation permissions, you can submit Spark jobs on Am -After you finish setting up the Lake Formation grants, you can [ submit Spark jobs on EMR Serverless.](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html#spark-examples) To run Iceberg jobs, you must provide the following `spark-submit` properties. - - - --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog - --conf spark.sql.catalog.spark_catalog.warehouse=<S3_DATA_LOCATION> - --conf spark.sql.catalog.spark_catalog.glue.account-id=<ACCOUNT_ID> - --conf spark.sql.catalog.spark_catalog.client.region=<REGION> - --conf spark.sql.catalog.spark_catalog.glue.endpoint=https://glue.<REGION>.amazonaws.com +After you finish setting up the Lake Formation grants, you can [ submit Spark jobs on EMR Serverless.](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-spark.html#spark-examples) The section that follows shows examples of how to configure and submit job run properties. @@ -193,0 +187,9 @@ Table maintenance and utility features | Not applicable +**Spark configuration for Iceberg:** The following sample shows how to configure Spark with Iceberg. To run Iceberg jobs, provide the following `spark-submit` properties. + + + --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog + --conf spark.sql.catalog.spark_catalog.warehouse=<S3_DATA_LOCATION> + --conf spark.sql.catalog.spark_catalog.glue.account-id=<ACCOUNT_ID> + --conf spark.sql.catalog.spark_catalog.client.region=<REGION> + --conf spark.sql.catalog.spark_catalog.glue.endpoint=https://glue.<REGION>.amazonaws.com + @@ -209,0 +212,29 @@ Table maintenance and utility features | Not supported +The following samples configure Spark with Hudi, specifying file locations and other properties necessary for use. + +**Spark config for Hudi:** This snippet when used in a notebook specifies the path to the Hudi Spark bundle JAR file, which enables Hudi functionality in Spark. It also configures Spark to use the AWS Glue Data Catalog as the metastore. + + + %%configure -f + { + "conf": { + "spark.jars": "/usr/lib/hudi/hudi-spark-bundle.jar", + "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory", + "spark.serializer": "org.apache.spark.serializer.JavaSerializer", + "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.hudi.catalog.HoodieCatalog", + "spark.sql.extensions": "org.apache.spark.sql.hudi.HoodieSparkSessionExtension" + } + } + +**Spark config for Hudi with AWS Glue:** This snippet when used in a notebook enables Hudi as a supported data-lake format and ensures that Hudi libraries and dependencies are available. + + + %%configure + { + "--conf": "spark.serializer=org.apache.spark.serializer.JavaSerializer --conf + spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog --conf + spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension", + "--datalake-formats": "hudi", + "--enable-glue-datacatalog": True, + "--enable-lakeformation-fine-grained-access": "true" + } + @@ -225,0 +257,7 @@ Table maintenance and utility features | Not supported +**EMR Serverless with Delta Lake:** To use Delta Lake with Lake Formation on EMR Serverless, run the following command: + + + spark-sql \ + --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,com.amazonaws.emr.recordserver.connector.spark.sql.RecordServerSQLExtension \ + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \ +