AWS emr documentation change
Summary
Added Hudi support documentation for Lake Formation Full Table Access (FTA), including configuration details, write support considerations, and a comprehensive operations support table.
Security assessment
The changes document security-related configurations for Hudi tables under Lake Formation's Full Table Access (security feature), including credential resolution via AWSLakeFormationCredentialResolver. However, there's no evidence of addressing a specific vulnerability or incident - this is routine documentation expansion for a new feature (Hudi FTA support).
Diff
diff --git a/emr/latest/EMR-Serverless-UserGuide/lake-formation-unfiltered-access.md b/emr/latest/EMR-Serverless-UserGuide/lake-formation-unfiltered-access.md index b59e2d22c..3c4c143ae 100644 --- a//emr/latest/EMR-Serverless-UserGuide/lake-formation-unfiltered-access.md +++ b//emr/latest/EMR-Serverless-UserGuide/lake-formation-unfiltered-access.md @@ -151,0 +152,15 @@ Delta Lake +Hudi + + + + ‐‐conf spark.hadoop.fs.s3.credentialsResolverClass=com.amazonaws.glue.accesscontrol.AWSLakeFormationCredentialResolver + --conf spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject=true + --conf spark.hadoop.fs.s3.folderObject.autoAction.disabled=true + --conf spark.sql.catalog.skipLocationValidationOnCreateTable.enabled=true + --conf spark.sql.catalog.createDirectoryAfterTable.enabled=true + --conf spark.sql.catalog.dropDirectoryBeforeTable.enabled=true + --conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar + --conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog + --conf spark.serializer=org.apache.spark.serializer.KryoSerializer + @@ -219,0 +235,20 @@ For Delta Lake tables +For Hudi tables + + + + %%configure -f + { + "conf": { + "spark.hadoop.fs.s3.credentialsResolverClass": "com.amazonaws.glue.accesscontrol.AWSLakeFormationCredentialResolver", + "spark.hadoop.fs.s3.useDirectoryHeaderAsFolderObject": true, + "spark.hadoop.fs.s3.folderObject.autoAction.disabled": true, + "spark.sql.catalog.skipLocationValidationOnCreateTable.enabled": true, + "spark.sql.catalog.createDirectoryAfterTable.enabled": true, + "spark.sql.catalog.dropDirectoryBeforeTable.enabled": true, + "spark.jars": "/usr/lib/hudi/hudi-spark-bundle.jar", + "spark.sql.extensions": "org.apache.spark.sql.hudi.HoodieSparkSessionExtension", + "spark.sql.catalog.spark_catalog": "org.apache.spark.sql.hudi.catalog.HoodieCatalog", + "spark.serializer": "org.apache.spark.serializer.KryoSerializer" + } + } + @@ -282 +317,11 @@ Operations not listed above will continue to use IAM permissions to access table - * Full Table Access is supported for Hive, Iceberg, and Delta tables. Hudi tables do not support full table access. + * Full Table Access is supported for Hive, Iceberg, Delta, and Hudi tables. + + * **Hudi FTA Write Support considerations:** + + * Hudi FTA writes require using HoodieCredentialedHadoopStorage for credential vending during job execution. Set the following configuration when running Hudi jobs: `hoodie.storage.class=org.apache.spark.sql.hudi.storage.HoodieCredentialedHadoopStorage` + + * Full Table Access (FTA) write support for Hudi is available starting from Amazon EMR release 7.12. + + * Hudi FTA write support currently works only with the default Hudi configurations. Custom or non-default Hudi settings may not be fully supported and could result in unexpected behavior. + + * Clustering for Hudi Merge-On-Read (MOR) tables is not supported at this point under FTA write mode. @@ -298,0 +344,76 @@ Operations not listed above will continue to use IAM permissions to access table +#### Hudi FTA Write Support - Supported Operations + +The following table shows the supported write operations for Hudi Copy-On-Write (COW) and Merge-On-Read (MOR) tables under Full Table Access mode: + +Hudi FTA Supported Write Operations Table Type | Operation | SQL Write Command | Status +---|---|---|--- +COW | INSERT | INSERT INTO TABLE | Supported +COW | INSERT | INSERT INTO TABLE - PARTITION (Static, Dynamic) | Supported +COW | INSERT | INSERT OVERWRITE | Supported +COW | INSERT | INSERT OVERWRITE - PARTITION (Static, Dynamic) | Supported +UPDATE | UPDATE | UPDATE TABLE | Supported +COW | UPDATE | UPDATE TABLE - Change Partition | Not Supported +DELETE | DELETE | DELETE FROM TABLE | Supported +ALTER | ALTER | ALTER TABLE - RENAME TO | Not Supported +COW | ALTER | ALTER TABLE - SET TBLPROPERTIES | Supported +COW | ALTER | ALTER TABLE - UNSET TBLPROPERTIES | Supported +COW | ALTER | ALTER TABLE - ALTER COLUMN | Supported +COW | ALTER | ALTER TABLE - ADD COLUMNS | Supported +COW | ALTER | ALTER TABLE - ADD PARTITION | Supported +COW | ALTER | ALTER TABLE - DROP PARTITION | Supported +COW | ALTER | ALTER TABLE - RECOVER PARTITIONS | Supported +COW | ALTER | REPAIR TABLE SYNC PARTITIONS | Supported +DROP | DROP | DROP TABLE | Supported +COW | DROP | DROP TABLE - PURGE | Supported +CREATE | CREATE | CREATE TABLE - Managed | Supported +COW | CREATE | CREATE TABLE - PARTITION BY | Supported +COW | CREATE | CREATE TABLE IF NOT EXISTS | Supported +COW | CREATE | CREATE TABLE LIKE | Supported +COW | CREATE | CREATE TABLE AS SELECT | Supported +CREATE | CREATE | CREATE TABLE with LOCATION - External Table | Not Supported +DATAFRAME(INSERT) | DATAFRAME(INSERT) | saveAsTable.Overwrite | Supported +COW | DATAFRAME(INSERT) | saveAsTable.Append | Not Supported +COW | DATAFRAME(INSERT) | saveAsTable.Ignore | Supported +COW | DATAFRAME(INSERT) | saveAsTable.ErrorIfExists | Supported +COW | DATAFRAME(INSERT) | saveAsTable - External table (Path) | Not Supported +COW | DATAFRAME(INSERT) | save(path) - DF v1 | Not Supported +MOR | INSERT | INSERT INTO TABLE | Supported +MOR | INSERT | INSERT INTO TABLE - PARTITION (Static, Dynamic) | Supported +MOR | INSERT | INSERT OVERWRITE | Supported +MOR | INSERT | INSERT OVERWRITE - PARTITION (Static, Dynamic) | Supported +UPDATE | UPDATE | UPDATE TABLE | Supported +MOR | UPDATE | UPDATE TABLE - Change Partition | Not Supported +DELETE | DELETE | DELETE FROM TABLE | Supported +ALTER | ALTER | ALTER TABLE - RENAME TO | Not Supported +MOR | ALTER | ALTER TABLE - SET TBLPROPERTIES | Supported +MOR | ALTER | ALTER TABLE - UNSET TBLPROPERTIES | Supported +MOR | ALTER | ALTER TABLE - ALTER COLUMN | Supported +MOR | ALTER | ALTER TABLE - ADD COLUMNS | Supported +MOR | ALTER | ALTER TABLE - ADD PARTITION | Supported +MOR | ALTER | ALTER TABLE - DROP PARTITION | Supported +MOR | ALTER | ALTER TABLE - RECOVER PARTITIONS | Supported +MOR | ALTER | REPAIR TABLE SYNC PARTITIONS | Supported +DROP | DROP | DROP TABLE | Supported +MOR | DROP | DROP TABLE - PURGE | Supported +CREATE | CREATE | CREATE TABLE - Managed | Supported +MOR | CREATE | CREATE TABLE - PARTITION BY | Supported +MOR | CREATE | CREATE TABLE IF NOT EXISTS | Supported +MOR | CREATE | CREATE TABLE LIKE | Supported +MOR | CREATE | CREATE TABLE AS SELECT | Supported +CREATE | CREATE | CREATE TABLE with LOCATION - External Table | Not Supported +DATAFRAME(UPSERT) | DATAFRAME(UPSERT) | saveAsTable.Overwrite | Supported +MOR | DATAFRAME(UPSERT) | saveAsTable.Append | Not Supported +MOR | DATAFRAME(UPSERT) | saveAsTable.Ignore | Supported +MOR | DATAFRAME(UPSERT) | saveAsTable.ErrorIfExists | Supported +MOR | DATAFRAME(UPSERT) | saveAsTable - External table (Path) | Not Supported +MOR | DATAFRAME(UPSERT) | save(path) - DF v1 | Not Supported +DATAFRAME(DELETE) | DATAFRAME(DELETE) | saveAsTable.Append | Not Supported +MOR | DATAFRAME(DELETE) | saveAsTable - External table (Path) | Not Supported +MOR | DATAFRAME(DELETE) | save(path) - DF v1 | Not Supported +DATAFRAME(BULK_INSERT) | DATAFRAME(BULK_INSERT) | saveAsTable.Overwrite | Supported +MOR | DATAFRAME(BULK_INSERT) | saveAsTable.Append | Not Supported +MOR | DATAFRAME(BULK_INSERT) | saveAsTable.Ignore | Supported +MOR | DATAFRAME(BULK_INSERT) | saveAsTable.ErrorIfExists | Supported +MOR | DATAFRAME(BULK_INSERT) | saveAsTable - External table (Path) | Not Supported +MOR | DATAFRAME(BULK_INSERT) | save(path) - DF v1 | Not Supported +