AWS lake-formation: Add access delegation modes and least-privilege IAM guidance for Databricks
Summary
Restructures the Databricks catalog federation guide: adds a new 'Change the access delegation mode' section describing source-vended (Databricks-issued, table-scoped S3 credentials) vs aws-vended (registered IAM role) modes, documents the CREDENTIAL_VENDING_MODE connection property, updates the IAM role trust policy to include lakeformation.amazonaws.com, states the minimum required Secrets Manager actions, notes that vended-credential mode needs no S3 permissions, and adds prerequisite/grant/query steps plus wording cleanups.
Security assessment
The diff documents credential-handling and IAM hardening for federated catalogs: it scopes the role trust policy to glue.amazonaws.com and lakeformation.amazonaws.com, calls out the minimum Secrets Manager actions (GetSecretValue/DescribeSecret/PutSecretValue) with no UpdateSecret or resource policy, and explains that source-vended mode issues per-table temporary S3 credentials so the registered role needs no S3 data access. This is security best-practice/least-privilege documentation rather than a fix for a specific vulnerability or incident, so it is not security_issue_related, but it is security documentation with IAM/credential impact (medium).
Evidence
+Create an IAM policy and attach it to an IAM role. Add both the AWS Glue and Lake Formation service principals to the trust policy, so a single IAM role serves both AWS Glue and Lake Formation. The secret actions shown (`GetSecretValue`, `DescribeSecret`, `PutSecretValue`) are the minimum required, and no secret resource policy or `UpdateSecret` permission is needed.
Diff
diff --git a/lake-formation/latest/dg/catalog-federation-databricks.md b/lake-formation/latest/dg/catalog-federation-databricks.md index 3c5665b..20593df 100644 --- a//lake-formation/latest/dg/catalog-federation-databricks.md +++ b//lake-formation/latest/dg/catalog-federation-databricks.md @@ -7 +7 @@ -PrerequisitesCreate Federated CatalogConsiderations when integrating with Databricks +PrerequisitesCreate a federated catalogChange the access delegation modeConsiderations when integrating with Databricks @@ -13 +13,3 @@ AWS Glue Data Catalog federates to Databricks using the OAuth2 credentials of a -Next, catalog federation enables discovery and query of Iceberg tables in your Databricks Unity Catalog. To read delta tables, please ensure Iceberg metadata is available for these tables using Uniform. Follow the Databricks tutorial and documentation to create the service principal and associated privileges in your Databricks workspace. +Catalog federation enables discovery and query of Iceberg tables in your Databricks Unity Catalog. To read delta tables, please ensure Iceberg metadata is available for these tables using Uniform. Follow the Databricks tutorial and documentation to create the service principal and associated privileges in your Databricks workspace. + +AWS Glue Data Catalog supports two access delegation modes, and you can configure the federated catalog to use credentials vended by either Databricks (source-vended) or the IAM role that Lake Formation uses (aws-vended). In the former mode, Databricks issues temporary Amazon S3 credentials scoped to each table, and AWS Lake Formation passes those credentials to your analytics engine. The IAM role you register with Lake Formation doesn't need access to the Amazon S3 buckets that store your Databricks tables. For a comparison of the two access delegation modes, see [Access delegation modes](./catalog-federation.html#access-delegation-modes). @@ -16,0 +19,17 @@ Next, catalog federation enables discovery and query of Iceberg tables in your D +Complete these steps in your Databricks workspace before you create the federated catalog. + + 1. **Make tables readable as Iceberg** + +Federation reads Iceberg metadata. For Delta tables, turn on Uniform. Column mapping is required for Iceberg compatibility, so set it in the same statement. + + 2. **Create a service principal and an OAuth secret** + +In the workspace settings, add a service principal, then generate an OAuth secret. Choose the API scopes the secret allows, such as `unity catalog sql`. Refer to the Databricks documentation for valid scope values. Record the application ID (client ID), the secret, and the scopes you chose. + + 3. **Ensure external data access is configured for the metastore** + + 4. **Grant Unity Catalog privileges to the service principal** + + + + @@ -32 +51 @@ Your IAM principal (user or role) must have the following permissions: -You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` permission on the Data Catalog +You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` permission on the Data Catalog. @@ -34 +53 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe -## Create Federated Catalog +## Create a federated catalog @@ -36 +55 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 1. Sign in to the the console and open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/). + 1. Sign in to the console and open the Lake Formation console at [https://console.aws.amazon.com/lakeformation/](https://console.aws.amazon.com/lakeformation/). @@ -40 +59 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 3. In the left navigation pane, choose Catalogs. + 3. In the navigation pane, choose **Catalogs**. @@ -44 +63 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 5. In **Choose data source** step, select Databricks from the available options. + 5. In the **Choose data source** step, choose Databricks Unity Catalog from the available options. @@ -46 +65 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 6. In **Set catalog details** step, you provide three information - catalog details, connection details, and registration details. + 6. In the **Set catalog details** step, you provide three types of information: catalog details, connection details, and registration details. @@ -48 +67 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 7. In **catalog details** container, provide a unique name to your AWS Glue federated catalog and enter the name of the existing Databricks catalog. + 7. In the **catalog details** container, provide a unique name for your AWS Glue federated catalog, and enter the name of the existing Databricks catalog. @@ -50 +69 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 8. In **connections details** container, you can either choose from an existing connection that you have access or provide configuration to create a new connector. + 8. In the **connections details** container, you can either choose an existing connection that you have access to, or provide configuration to create a new connector. @@ -58 +77 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - * Authentication – Specify the authentication configuration that AWS Glue uses to connect to remote catalog server. AWS Glue supports both OAuth2 and Custom authentication. + * Authentication – Specify the authentication configuration that AWS Glue uses to connect to the remote catalog server. AWS Glue supports both OAuth2 and Custom authentication. @@ -60 +79 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - * Token URL – Specify the URL of remote catalog's identity provider. + * Token URL – Specify the URL of the remote catalog's identity provider. @@ -64 +83 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - * Secret – Store and use OAuth2 client secret using AWS Secrets Manager or enter the secret value in textbox. When you enter the secret manually in console, AWS Glue creates the secret on your behalf. + * Secret – Store and use the OAuth2 client secret by using AWS Secrets Manager, or enter the secret value in the text box. When you enter the secret manually in the console, AWS Glue creates the secret on your behalf. @@ -68 +87,3 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 10. Create an IAM role that AWS Glue and Lake Formation service principals can use to access secret in and Amazon S3 locations of remote Iceberg tables respectively. Select the IAM role in the registration dropdown. Refer to step 2 and 3 in following CLI section for IAM policy details. + 10. Under **Access delegation mode** , choose **Vended credential** to use credentials issued by Databricks. Choose **Registered role** to have Lake Formation issue credentials using your IAM role. + + 11. In **Register Glue connection with Lake Formation** , choose how to provide the IAM role: @@ -70 +91 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 11. Select Test Connection to test whether your connection properties and IAM role access are configured correctly. Test connection functionality is not available when connecting to Databricks using Amazon VPC. + * **Create and use a default role** – Lake Formation creates the role when you create the catalog. The console shows the role name, `LakeFormationIRC-<connection-name>-<suffix>`, before you create. The role's permissions policy allows `secretsmanager:GetSecretValue`, `secretsmanager:DescribeSecret`, and `secretsmanager:PutSecretValue` on the selected secret only. Its trust policy lets only AWS Glue and Lake Formation in this account and Region assume the role. @@ -72 +93 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 12. Select **Next** to review your settings. + * **Use an existing role** – Choose a role that you created. For the required policies, see step 2 in Using the AWS CLI, and step 3 if you choose **Registered role**. With **Vended credential** , the role needs access only to the secret (and to the Amazon VPC network interfaces, if you use Amazon VPC). It doesn't need Amazon S3 permissions. @@ -74 +95,5 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe - 13. Select **Create Catalog** in review page. + 12. (Optional) Choose **Run test** to confirm your connection properties and IAM role access are configured correctly. A successful test means AWS Glue exchanged the client secret for a token and reached the Databricks Iceberg REST endpoint. **Run test** isn't available when you use the default role that Lake Formation creates for you or when you are using Amazon VPC to connect to Databricks. + + 13. Choose **Next** to review your settings. + + 14. Choose **Create Catalog** on the review page. The console creates the AWS Glue connection, registers it with Lake Formation, and creates the federated catalog in a single step. @@ -81 +106 @@ You must be a Lake Formation data lake administrator or have `CREATE_CATALOG` pe -The AWS Glue connector supports two authentication types - **OAuth2** and **Custom**. When using OAuth2 option, use AWS Secrets Manager to store client secret of the Databricks service principal. You will later use this secret when creating the AWS Glue Connection. For Custom authentication, use AWS Secrets Manager to store and retrieve the access token. +The AWS Glue connector supports two authentication types: **OAuth2** and **Custom**. When you use the OAuth2 option, use AWS Secrets Manager to store the client secret of the Databricks service principal. You use this secret later when you create the AWS Glue connection. For Custom authentication, use AWS Secrets Manager to store and retrieve the access token. @@ -95 +120 @@ In the following example, replace ``<databricks-secret>``, ``<client_secret>``, -`USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET` is a reserved keyword that AWS Glue uses to refer to a client secret value in the secret. Use the same keyword when you are creating the secret in Lake Formation console too. +`USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET` is a reserved keyword that AWS Glue uses to refer to a client secret value in the secret. Use the same keyword when you create the secret in the Lake Formation console. AWS Glue adds and refreshes the OAuth token keys in the secret itself, so you don't need to create placeholder token keys. @@ -97 +122 @@ In the following example, replace ``<databricks-secret>``, ``<client_secret>``, - 2. **Create an IAM role which gives AWS Glue connection object access to secret created in previous step** + 2. **Create an IAM role that gives the AWS Glue connection object access to the secret created in the previous step** @@ -101 +126 @@ The AWS Glue connection object requires access to the AWS Secrets Manager secret -Create an IAM policy and attach it to an IAM role. Add AWS Glue service principal to the trust policy. +Create an IAM policy and attach it to an IAM role. Add both the AWS Glue and Lake Formation service principals to the trust policy, so a single IAM role serves both AWS Glue and Lake Formation. The secret actions shown (`GetSecretValue`, `DescribeSecret`, `PutSecretValue`) are the minimum required, and no secret resource policy or `UpdateSecret` permission is needed. @@ -145 +170,4 @@ In the following example, replace ``<your-secrets-manager-ARN>``, ``<your-vpc-id - "Service": "glue.amazonaws.com" + "Service": [ + "glue.amazonaws.com", + "lakeformation.amazonaws.com" + ] @@ -151 +179 @@ In the following example, replace ``<your-secrets-manager-ARN>``, ``<your-vpc-id - 3. **Create an IAM policy that gives Lake Formation read access to catalog's Amazon S3 location** + 3. Skip this step if you use vended-credential mode. In that mode, the remote catalog issues the Amazon S3 credentials, so the IAM role doesn't need access to the table data in Amazon S3. @@ -153 +181 @@ In the following example, replace ``<your-secrets-manager-ARN>``, ``<your-vpc-id -As the catalog owner of a federated catalog in Data Catalog, you use Lake Formation to grant coarse-grained table access, fine-grained - column-level, row-level, and cell-level - access, and tag-based access to your data teams. Lake Formation uses an IAM role that gives it access to the underlying Amazon S3 locations of your remote Iceberg tables. This access allows Lake Formation to vend scoped access credentials to analytics engines querying remote tables. +**Create an IAM policy that gives Lake Formation read access to the catalog's Amazon S3 location** @@ -155 +183,3 @@ As the catalog owner of a federated catalog in Data Catalog, you use Lake Format -Create IAM policy and attach to an IAM role. Add Lake Formation service principal to the IAM role trust policy. +As the catalog owner of a federated catalog in Data Catalog, you use Lake Formation to grant coarse-grained table access, fine-grained access (column-level, row-level, and cell-level), and tag-based access to your data teams. Lake Formation uses an IAM role that gives it access to the underlying Amazon S3 locations of your remote Iceberg tables. With this access, Lake Formation can vend scoped access credentials to the analytics engines that query remote tables. + +Create an IAM policy and attach it to an IAM role. Add the Lake Formation service principal to the IAM role trust policy. @@ -212 +242 @@ In the following example, replace ``<your-s3-bucket-N>`` and ``<your-kms-key>`` -When you use Lake Formation console to create a federated catalog, the console uses a single IAM role with both policies attached to complete setup. +When you use the Lake Formation console to create a federated catalog, the console uses a single IAM role with both policies attached to complete setup. @@ -218 +248 @@ Data Catalog supports connectionType: `DATABRICKSICEBERGRESTCATALOG` for connect -The following example uses OAuth2 authentication configuration to create an AWS Glue connection. Replace ``highlighed sections`` with your information. +The following example uses OAuth2 authentication configuration to create an AWS Glue connection. Replace ``highlighted sections`` with your information. @@ -225,2 +255,3 @@ The following example uses OAuth2 authentication configuration to create an AWS - "INSTANCE_URL": "<your-databricks-workspace-catalog-URL>", - "ROLE_ARN": "<your-IAM-role-for-secrets-and-VPC-access>" + "INSTANCE_URL": "https://<workspace-host>/api/2.1/unity-catalog/iceberg-rest", + "ROLE_ARN": "<your-IAM-role-for-secrets-and-VPC-access>", + "CREDENTIAL_VENDING_MODE": "SOURCE_VENDED" @@ -237 +268 @@ The following example uses OAuth2 authentication configuration to create an AWS - "scope": "all-apis" + "scope": "<scopes-assigned-to-the-secret>" @@ -244 +275,3 @@ The following example uses OAuth2 authentication configuration to create an AWS - 5. **Register AWS Glue connection as a Lake Formation resource** +The console accepts the workspace root URL and stores it with the `/api/2.1/unity-catalog/iceberg-rest` path appended; in the CLI, pass the full path. The console might also set `CATALOG_CASING_FILTER` (for example, `LOWERCASE_ONLY`). + + 5. **Register the AWS Glue connection as a Lake Formation resource** @@ -246 +279 @@ The following example uses OAuth2 authentication configuration to create an AWS -Using the AWS Glue connection object (created in Step 4) and IAM role (created in Step 3), you can now register the AWS Glue connection object as a Lake Formation managed resource. +Using the AWS Glue connection object (created in Step 4) and the IAM role (created in Step 3), you can now register the AWS Glue connection object as a Lake Formation managed resource. @@ -255,0 +289,6 @@ Replace ``<your-glue-connector-arn>`` and ``<your-IAM-role-ARN-having-LF-access> +The `--with-federation` flag is required. The `--with-privileged-access` flag is optional: per the RegisterResource API reference, it gives the calling principal permission to perform all supported Lake Formation operations on the registered resource. It does not affect how credentials are vended, and the console sets it by default. + +###### Note + +Register the connection before you create the catalog. Otherwise, `create-catalog` fails with `The data source … is not registered`. + @@ -260 +299 @@ After creating an AWS Glue connection object and registering it with Lake Format -Provide the federated catalog a unique name at ``<your-federated-catalog-name>``, reference the catalog in Databricks at ``"<catalog-name-in-Databricks>``, and input connection name created earlier at ``"<your-glue-connection-name>``. +Provide a unique name for the federated catalog at ``<your-federated-catalog-name>``, reference the catalog in Databricks at ``"<catalog-name-in-Databricks>``, and enter the connection name you created earlier at ``"<your-glue-connection-name>``. @@ -272,0 +312,16 @@ Provide the federated catalog a unique name at ``<your-federated-catalog-name>`` + 7. **Grant Lake Formation permissions** + +Catalog-level grants don't carry down to federated databases and tables. Grant `DESCRIBE` on the database, and `SELECT` and `DESCRIBE` on the table wildcard, to each principal that queries the catalog. + + 8. **Query the federated catalog** + +Query the federated catalog from your analytics engine. For example, in Amazon Athena: + + SELECT * FROM "<federated-catalog>"."<schema>"."<table>" LIMIT 10; + + + + +## Change the access delegation mode + +To switch an existing connection between modes, update the connection property; you don't need to recreate the federated catalog. @@ -274,0 +330,3 @@ Provide the federated catalog a unique name at ``<your-federated-catalog-name>`` + aws glue get-connection --name <connection-name> --output json > connection.json + # Build ConnectionInput from connection.json with CREDENTIAL_VENDING_MODE set to the new value + aws glue update-connection --name <connection-name> --connection-input file://<updated-connection-input>.json @@ -278 +336 @@ Provide the federated catalog a unique name at ``<your-federated-catalog-name>`` - * When you drop resources (like databases and tables) in the Databricks, Lake Formation does not automatically revoke the permissions granted on that federated resource. To remove the access permissions, you need to explicitly revoke the permissions that were previously granted on the federated resource using Lake Formation. + * When you drop resources (such as databases and tables) in Databricks, Lake Formation does not automatically revoke the permissions granted on that federated resource. To remove the access permissions, you need to explicitly revoke the permissions that you previously granted on the federated resource by using Lake Formation. @@ -280 +338 @@ Provide the federated catalog a unique name at ``<your-federated-catalog-name>`` - * You can query Iceberg tables stored in Amazon S3 using this integration. When using any other table format or object storage, you can federate metadata in remote catalogs to AWS Glue and list its databases and tables but query operations like `SELECT ColumnFoo from TableBar` will fail during query with error 'Failed to read Apache Iceberg table. Object storage location is not supported.' + * You can query Iceberg tables stored in Amazon S3 by using this integration. When you use any other table format or object storage, you can federate metadata in remote catalogs to AWS Glue and list its databases and tables. However, query operations such as `SELECT ColumnFoo from TableBar` fail during the query with the error 'Failed to read Apache Iceberg table. Object storage location is not supported.' @@ -282 +340 @@ Provide the federated catalog a unique name at ``<your-federated-catalog-name>`` - * You can reuse the same AWS Glue connection to create multiple federated catalogs. Deleting a catalog will not delete the associated connection object. To delete a connection object, please use AWS CLI `aws glue delete-connection` command and ensure all associated catalogs are deleted first. + * You can reuse the same AWS Glue connection to create multiple federated catalogs. Deleting a catalog does not delete the associated connection object. To delete a connection object, use the AWS CLI `aws glue delete-connection` command, and make sure that you delete all associated catalogs first.