AWS emr high security documentation change
Summary
Updated limitations and security controls for Amazon EMR with Lake Formation, including restricted access to system driver logs, encryption requirements, supported access controls, and Iceberg table security measures.
Security assessment
The changes explicitly restrict access to system driver logs containing sensitive information, enforce encryption for logs, and limit functionalities that could compromise isolation (e.g., blocking UDFs/custom data sources). These updates address potential security risks like unauthorized data exposure or privilege escalation via logs or untrusted code execution. The addition of log encryption and access restrictions directly mitigates security vulnerabilities.
Diff
diff --git a/emr/latest/ManagementGuide/emr-lf-limitations.md b/emr/latest/ManagementGuide/emr-lf-limitations.md index caaa0c972..89138cfff 100644 --- a//emr/latest/ManagementGuide/emr-lf-limitations.md +++ b//emr/latest/ManagementGuide/emr-lf-limitations.md @@ -9 +9 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * [Table-level access control](./emr-lf-enable.html#emr-lf-table-perms) is available on clusters with Amazon EMR releases 6.13 and higher. +Amazon EMR with Lake Formation is available in all [available regions](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-region.html). @@ -11 +11 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * [Fine-grained access control](./emr-lf-enable.html#emr-lf-fgac-perms) at row, column, and cell level is available on clusters with Amazon EMR releases 6.15 and higher. + * Amazon EMR supports fine-grained access control via Lake Formation only for Apache Hive and Apache Iceberg tables. Apache Hive formats include Parquet, ORC, and xSV. @@ -13 +13 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Users with access to a table can access all the properties of that table. If you have Lake Formation based access control on a table, review the table to make sure that the properties don't contain any sensitive data or information. + * You can't turn off `DynamicResourceAllocation` for Lake Formation jobs. @@ -15 +15 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Amazon EMR clusters with Lake Formation don't support Spark's fallback to HDFS when Spark collects table statistics. This ordinarily helps optimize query performance. + * You can only use Lake Formation with Spark jobs. @@ -17 +17 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Operations that support access controls based on Lake Formation with non-governed Apache Spark tables include `INSERT INTO` and `INSERT OVERWRITE`. + * Amazon EMR with Lake Formation only supports a single Spark session throughout a job. @@ -19 +19 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Operations that support access controls based on Lake Formation with Apache Spark and Apache Hive include `SELECT`, `DESCRIBE`, `SHOW DATABASE`, `SHOW TABLE`, `SHOW COLUMN`, and `SHOW PARTITION`. + * Amazon EMR with Lake Formation only supports cross-account table queries shared through resource links. @@ -21 +21 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Amazon EMR doesn't support access control to the following Lake Formation based operations: + * The following aren't supported: @@ -23 +23 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Writes to governed tables + * Resilient distributed datasets (RDD) @@ -25 +25 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Amazon EMR doesn't support `CREATE TABLE`. Amazon EMR 6.10.0 and higher supports `ALTER TABLE`. + * Spark streaming @@ -27 +27 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * DML statements other than `INSERT` commands. + * Write with Lake Formation granted permissions @@ -29 +29 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * There are performance differences between the same query with and without Lake Formation based access control. + * Access control for nested columns @@ -31 +31 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * You can only use Amazon EMR with Lake Formation for Spark jobs. + * Amazon EMR blocks functionalities that might undermine the complete isolation of system driver, including the following: @@ -33 +33,47 @@ Consider the following when using Amazon EMR with AWS Lake Formation. - * Trusted Identity propagation is not supported with multi-catalog hierarchy in Glue Data Catalog. For more information, see [Working with a multi-catalog hierarchy in AWS Glue Data Catalog](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-multi-catalog.html). + * UDTs, HiveUDFs, and any user-defined function that involves custom classes + + * Custom data sources + + * Supply of additional jars for Spark extension, connector, or metastore + + * `ANALYZE TABLE` command + + * To enforce access controls, `EXPLAIN PLAN` and DDL operations such as `DESCRIBE TABLE` don't expose restricted information. + + * Amazon EMR restricts access to system driver Spark logs on Lake Formation-enabled applications. Since the system driver runs with elevated permissions, events and logs that the system driver generates can include sensitive information. To prevent unauthorized users or code from accessing this sensitive data, Amazon EMR disables access to system driver logs. + +System profile logs are always persisted in managed storage – this is a mandatory setting that cannot be disabled. These logs are stored securely and encrypted using either a Customer Managed KMS key or an AWS Managed KMS key. + +If your Amazon EMR application is in a private subnet with VPC endpoints for Amazon S3 and you attach an endpoint policy to control access, before your jobs can send log data to AWS Managed Amazon S3, you must include the permissions detailed in [Managed storage](logging.html#jobs-log-storage-managed-storage) in your VPC policy to S3 gateway endpoint. For troubleshooting requests, contact AWS support. + + * If you registered a table location with Lake Formation, the data access path goes through the Lake Formation stored credentials regardless of the IAM permission for the Amazon EMR job runtime role. If you misconfigure the role registered with table location, jobs submitted that use the role with S3 IAM permission to the table location will fail. + + * Writing to a Lake Formation table uses IAM permission rather than Lake Formation granted permissions. If your job runtime role has the necessary S3 permissions, you can use it to run write operations. + + + + +The following are considerations and limitations when using Apache Iceberg: + + * You can only use Apache Iceberg with session catalog and not arbitrarily named catalogs. + + * Iceberg tables that are registered in Lake Formation only support the metadata tables `history`, `metadata_log_entries`, `snapshots`, `files`, `manifests`, and `refs`. Amazon EMR hides the columns that might have sensitive data, such as `partitions`, `path`, and `summaries`. This limitation doesn't apply to Iceberg tables that aren't registered in Lake Formation. + + * Tables that you don't register in Lake Formation support all Iceberg stored procedures. The `register_table` and `migrate` procedures aren't supported for any tables. + + * We recommend that you use Iceberg DataFrameWriterV2 instead of V1. + + * EMR 7.10 provides a way to switch back to RecordServer if you would like to use features supported by RecordServer, but not yet supported by native FGAC, such as writeback to Lake Formation registered tables. To switch back, specify the following configurations when launching the cluster. + + { + "Classification": "spark-defaults", + "Properties": { + "spark.emr.lakeformation.legacy.enabled": "true" + } + }, + { + "Classification": "yarn-site", + "Properties": { + "spark.emr.lakeformation.legacy.enabled": "true" + } + }