AWS emr documentation change
Summary
Restructured documentation into sections (General, Permissions, Logs and debugging, Iceberg) with expanded details about Lake Formation permissions, logging encryption, Iceberg limitations, and Spark operations
Security assessment
Added documentation about security features including system driver log restrictions to prevent unauthorized access to sensitive data, mandatory encryption of logs with KMS keys, and clarification of Lake Formation credential enforcement. However, no specific security vulnerability or incident remediation is mentioned.
Diff
diff --git a/emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable-considerations.md b/emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable-considerations.md index 9558d8a93..1dd33cdac 100644 --- a//emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable-considerations.md +++ b//emr/latest/EMR-Serverless-UserGuide/emr-serverless-lf-enable-considerations.md @@ -4,0 +5,2 @@ +GeneralPermissionsLogs and debuggingIceberg + @@ -7 +9,3 @@ -Consider the following considerations and limitations when you use Lake Formation with EMR Serverless. +## General + +Review the following limitations when using Lake Formation with EMR Serverless. @@ -13 +17,3 @@ When you enable Lake Formation for a Spark job on EMR Serverless, the job launch -Amazon EMR Serverless with Lake Formation is available in all supported [EMR Serverless Regions](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/considerations.html). + * Amazon EMR Serverless with Lake Formation is available in all supported [EMR Serverless Regions](https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/considerations.html). + + @@ -15 +20,0 @@ Amazon EMR Serverless with Lake Formation is available in all supported [EMR Ser - * Amazon EMR Serverless supports fine-grained access control via Lake Formation for read operations with Apache Hive, Apache Iceberg, Delta Lake and Hudi tables. Apache Hive formats include Parquet, ORC, and xSV. @@ -33,2 +37,0 @@ Amazon EMR Serverless with Lake Formation is available in all supported [EMR Ser - * Write with Lake Formation granted permissions - @@ -47 +50 @@ Amazon EMR Serverless with Lake Formation is available in all supported [EMR Ser - * To enforce access controls, `EXPLAIN PLAN` and DDL operations such as `DESCRIBE TABLE` don't expose restricted information. + * If your EMR Serverless application is in a private subnet with VPC endpoints for Amazon S3 and you attach an endpoint policy to control access, before your jobs can send log data to AWS Managed Amazon S3, include the permissions detailed in [Managed storage](logging.html#jobs-log-storage-managed-storage) in your VPC policy to S3 gateway endpoint. For troubleshooting requests, contact AWS support. @@ -49 +52 @@ Amazon EMR Serverless with Lake Formation is available in all supported [EMR Ser - * EMR Serverless restricts access to system driver Spark logs on Lake Formation-enabled applications. Since the system driver runs with elevated permissions, events and logs that the system driver generates can include sensitive information. To prevent unauthorized users or code from accessing this sensitive data, EMR Serverless disables access to system driver logs. + * Starting with Amazon EMR 7.9.0, Spark FGAC supports S3AFileSystem when used with the s3a:// scheme. @@ -51 +54 @@ Amazon EMR Serverless with Lake Formation is available in all supported [EMR Ser -System profile logs are always persisted in managed storage – this is a mandatory setting that cannot be disabled. These logs are stored securely and encrypted using either a Customer Managed KMS key or an AWS Managed KMS key. + * Amazon EMR 7.11 supports creating managed tables using CTAS. @@ -53 +56 @@ System profile logs are always persisted in managed storage – this is a mandat -If your EMR Serverless application is in a private subnet with VPC endpoints for Amazon S3 and you attach an endpoint policy to control access, before your jobs can send log data to AWS Managed Amazon S3, include the permissions detailed in [Managed storage](logging.html#jobs-log-storage-managed-storage) in your VPC policy to S3 gateway endpoint. For troubleshooting requests, contact AWS support. + * Amazon EMR 7.12 supports creating managed and external tables using CTAS. @@ -55 +57,0 @@ If your EMR Serverless application is in a private subnet with VPC endpoints for - * If you registered a table location with Lake Formation, the data access path goes through the Lake Formation stored credentials regardless of the IAM permission for the EMR Serverless job runtime role. If you misconfigure the role registered with table location, jobs submitted that use the role with S3 IAM permission to the table location will fail. @@ -57 +58,0 @@ If your EMR Serverless application is in a private subnet with VPC endpoints for - * Writing to a Lake Formation table uses IAM permission rather than Lake Formation granted permissions. If your job runtime role has the necessary S3 permissions, you can use it to run write operations. @@ -59 +60,32 @@ If your EMR Serverless application is in a private subnet with VPC endpoints for - * Starting with Amazon EMR 7.9.0, Spark FGAC supports S3AFileSystem when used with the s3a:// scheme. + +## Permissions + + * To enforce access controls, EXPLAIN PLAN and DDL operations such as DESCRIBE TABLE don't expose restricted information. + + * When you register a table location with Lake Formation, data access uses Lake Formation stored credentials instead of the EMR Serverless job runtime role's IAM permissions. Jobs will fail if the registered role for table location is misconfigured, even when the runtime role has S3 IAM permissions for that location. + + * Starting with Amazon EMR 7.12, you can write to existing Hive and Iceberg tables using DataFrameWriter (V2) with Lake Formation credentials in append mode. For overwrite operations or when creating new tables, EMR uses the runtime role credentials to modify table data. + + * The following limitations apply when using views or cached tables as source data (these limitations do not apply to AWS Glue Data Catalog views): + + * For MERGE, DELETE, and UPDATE operations + + * Supported: Using views and cached tables as source tables. + + * Not supported: Using views and cached tables in assignment and condition clauses. + + * For CREATE OR REPLACE and REPLACE TABLE AS SELECT operations: + + * Not supported: Using views and cached tables as source tables. + + * Delta Lake tables with UDFs in source data support MERGE, DELETE, and UPDATE operations only when deletion vector is enabled. + + + + +## Logs and debugging + + * EMR Serverless restricts access to system driver Spark logs on Lake Formation-enabled applications. Since the system driver runs with elevated permissions, events and logs that the system driver generates can include sensitive information. To prevent unauthorized users or code from accessing this sensitive data, EMR Serverless disables access to system driver logs. + + * System profile logs are always persisted in managed storage – this is a mandatory setting that cannot be disabled. These logs are stored securely and encrypted using either a Customer Managed KMS key or an AWS Managed KMS keys. + @@ -62,0 +95 @@ If your EMR Serverless application is in a private subnet with VPC endpoints for +## Iceberg @@ -64 +97 @@ If your EMR Serverless application is in a private subnet with VPC endpoints for -The following are considerations and limitations when using Apache Iceberg: +Review the following considerations when using Apache Iceberg: @@ -70 +103 @@ The following are considerations and limitations when using Apache Iceberg: - * Tables that you don't register in Lake Formation support all Iceberg stored procedures. The `register_table` and `migrate` procedures aren't supported for any tables. + * Tables that not registered in Lake Formation support all Iceberg stored procedures. The `register_table` and `migrate` procedures aren't supported for any tables.