AWS glue documentation change
Summary
Restructured the document by removing detailed content and replacing it with topic links. The change reorganizes the content into a more modular format with separate pages for each topic.
Security assessment
The changes are purely organizational, moving detailed content to separate pages and replacing it with topic links. There is no evidence of security-related content being added or modified.
Diff
diff --git a/glue/latest/dg/access_catalog.md index e3d2ed22d..752c9dbc3 100644 --- a/glue/latest/dg/access_catalog.md +++ b/glue/latest/dg/access_catalog.md @@ -5,2 +4,0 @@ -Connecting to the Data Catalog using AWS Glue Iceberg REST endpointConnecting to the Data Catalog using AWS Glue Iceberg REST extension endpoint Authenticating and authorizing access to AWS Glue service endpoints Connecting to Data Catalog from a standalone Spark applicationData mapping between Amazon Redshift and Apache IcebergConsiderations and limitations when using AWS Glue Iceberg REST Catalog APIs - @@ -24,218 +22 @@ You can use the AWS Glue Data Catalog (Data Catalog) to discover and understand -## Connecting to the Data Catalog using AWS Glue Iceberg REST endpoint - -AWS Glue's Iceberg REST endpoint supports API operations specified in the Apache Iceberg REST specification. Using an Iceberg REST client, you can connect your application running on an analytics engine to the REST catalog hosted in the Data Catalog. - -The endpoint supports both Apache Iceberg table specifications - v1 and v2, defaulting to v2. When using the Iceberg table v1 specification, you must specify v1 in the API call. Using the API operation, you can access Iceberg tables stored in both Amazon S3 object storage and Amazon S3 Table storage. - -**Endpoint configuration** - -You can access the AWS Glue Iceberg REST catalog using the service endpoint. Refer to the [AWS Glue service endpoints reference guide](https://docs.aws.amazon.com/general/latest/gr/glue.html#glue_region) for the region-specific endpoint. For example, when connecting to AWS Glue in the us-east-1 Region, you need to configure the endpoint URI property as follows: - - - Endpoint : https://glue.us-east-1.amazonaws.com/iceberg - -**Additional configuration properties** – When using Iceberg client to connect an analytics engine like Spark to the service endpoint, you are required to specify the following application configuration properties: - - - catalog_name = "mydatacatalog" - aws_account_id = "123456789012" - aws_region = "us-east-1" - spark = SparkSession.builder \ - ... \ - .config("spark.sql.defaultCatalog", catalog_name) \ - .config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") \ - .config(f"spark.sql.catalog.{catalog_name}.type", "rest") \ - .config(f"spark.sql.catalog.{catalog_name}.uri", "https://glue.{aws_region}.amazonaws.com/iceberg") \ - .config(f"spark.sql.catalog.{catalog_name}.warehouse", "{aws_account_id}") \ - .config(f"spark.sql.catalog.{catalog_name}.rest.sigv4-enabled", "true") \ - .config(f"spark.sql.catalog.{catalog_name}.rest.signing-name", "glue") \ - .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ - .getOrCreate() - - -## Connecting to the Data Catalog using AWS Glue Iceberg REST extension endpoint - -AWS Glue Iceberg REST extension endpoint provides additional APIs, which are not present in the Apache Iceberg REST specification, and provides server-side scan planning capabilities. These additional APIs are used when you access tables stored in Amazon Redshift managed storage. The endpoint is accessible from an application using Apache Iceberg AWS Glue Data Catalog extensions. - -**Endpoint configuration** -– A catalog with tables in the Redshift managed storage is accessible using the service endpoint. Refer to the [AWS Glue service endpoints reference guide](https://docs.aws.amazon.com/general/latest/gr/glue.html#glue_region) for the region-specific endpoint. For example, when connecting to AWS Glue in the us-east-1 Region, you need to configure the endpoint URI property as follows: - - - Endpoint : https://glue.us-east-1.amazonaws.com/extensions - - - catalog_name = "myredshiftcatalog" - aws_account_id = "123456789012" - aws_region = "us-east-1" - spark = SparkSession.builder \ - .config("spark.sql.defaultCatalog", catalog_name) \ - .config(f"spark.sql.catalog.{catalog_name}", "org.apache.iceberg.spark.SparkCatalog") \ - .config(f"spark.sql.catalog.{catalog_name}.type", "glue") \ - .config(f"spark.sql.catalog.{catalog_name}.glue.id", "{123456789012}:redshiftnamespacecatalog/redshiftdb") \ - .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ - .getOrCreate() - - -## Authenticating and authorizing access to AWS Glue service endpoints - -API requests to the AWS Glue Data Catalog endpoints are authenticated using AWS Signature Version 4 (SigV4). See [AWS Signature Version 4 for API requests](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv.html) section to learn more about AWS SigV4. - -When accessing the AWS Glue service endpoint and AWS Glue metadata, the application assumes an IAM role which requires `glue:getCatalog` IAM action. - -**REST operation** | **REST path** | **AWS Glue IAM Action** | **CloudTrail EventName** | **Lake Formation permissions** ----|---|---|---|--- -GetConfig | GET /config | GetCatalog | GetConfig | Not required. -ListNamespaces | GET /namespaces | GetDatabases | GetDatabases | ALL, DESCRIBE, SELECT -CreateNamespace | POST /namespaces | CreateDatabase | CreateDatabase | ALL, CREATE_DATABASE -LoadNamespaceMetadata | GET /namespaces/{ns} | GetDatabase | GetDatabase | ALL, DESCRIBE, SELECT -UpdateProperties | POST /namespaces/{ns}/properties | UpdateDatabase | UpdateDatabase | ALL, ALTER -DeleteNamespace | DELETE /namespace/{ns} | DeleteDatabase | DeleteDatabase | ALL, DROP -ListTables | GET /namespaces/{ns}/tables | GetTables | GetTables | ALL, SELECT, DESCRIBE -CreateTable | POST /namespaces/{ns}/tables | CreateTable | CreateTable | ALL, CREATE_TABLE -LoadTable | GET /namespaces/{ns}/tables/{tbl} | GetTable | GetTable | ALL, SELECT, DESCRIBE -TableExists | HEAD /namespaces/{ns}/tables/{tbl} | GetTable | GetTable | ALL, SELECT, DESCRIBE -UpdateTable | POST /namespaces/{ns}/tables/{tbl} | UpdateTable | UpdateTable | ALL, ALTER -DeleteTable | DELETE /namespaces/{ns}/tables/{tbl} | DeleteTable | DeleteTable | ALL, DROP - -You can use IAM, AWS Lake Formation, or Lake Formation hybrid mode permissions to manage access to the default Data Catalog and its objects. - -Federated catalogs in the AWS Glue Data Catalog have Lake Formation registered data locations. Lake Formationintegrates with Data Catalog and provides database-style permissions to manage user access to catalog objects. In Lake Formation, permissions must be set up for the IAM user or role that is used to create, insert, or delete data. The permissions are the same as for existing AWS Glue tables: - - * CREATE_CATALOG – Required to create catalogs - - * CREATE_DATABASE – Required to create databases - - * CREATE_TABLE – Required to create tables - - * DELETE – Required to delete data from a table - - * DESCRIBE – Required to read metadata - - * DROP – Required to drop/delete a table or database - - * INSERT: Needed when the principal needs to insert data into a table - - * SELECT: Needed when the principal needs to select data from a table - - - - -For more information, see [Lake Formation permissions reference](lake-formation/latest/dg/lf-permissions-reference.html) in the AWS Lake Formation Developer Guide. - -## Connecting to Data Catalog from a standalone Spark application - -You can connect to the Data Catalog from a stand application using an Apache Iceberg connector. - - 1. Create an IAM role for Spark application. - - 2. Connect to AWS Glue Iceberg Rest endpoint using Iceberg connector. - - # configure your application. Refer to https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html for best practices on configuring environment variables. - export AWS_ACCESS_KEY_ID=$(aws configure get appUser.aws_access_key_id) - export AWS_SECRET_ACCESS_KEY=$(aws configure get appUser.aws_secret_access_key) - export AWS_SESSION_TOKEN=$(aws configure get appUser.aws_secret_token) - - export AWS_REGION=us-east-1 - export REGION=us-east-1 - export AWS_ACCOUNT_ID = {specify your aws account id here} - - ~/spark-3.5.3-bin-hadoop3/bin/spark-shell \ - --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.6.0 \ - --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" \ - --conf "spark.sql.defaultCatalog=spark_catalog" \ - --conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkCatalog" \ - --conf "spark.sql.catalog.spark_catalog.type=rest" \ - --conf "spark.sql.catalog.spark_catalog.uri=https://glue.us-east-1.amazonaws.com/iceberg" \ - --conf "spark.sql.catalog.spark_catalog.warehouse = {AWS_ACCOUNT_ID}" \ - --conf "spark.sql.catalog.spark_catalog.rest.sigv4-enabled=true" \ - --conf "spark.sql.catalog.spark_catalog.rest.signing-name=glue" \ - --conf "spark.sql.catalog.spark_catalog.rest.signing-region=us-east-1" \ - --conf "spark.sql.catalog.spark_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO" \ - --conf "spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialProvider" - - - 3. Query data in the Data Catalog. - - spark.sql("create database myicebergdb").show() - spark.sql("""CREATE TABLE myicebergdb.mytbl (name string) USING iceberg location 's3://bucket_name/mytbl'""") - spark.sql("insert into myicebergdb.mytbl values('demo') ").show() - - - - - -## Data mapping between Amazon Redshift and Apache Iceberg - -Redshift and Iceberg support various data types. The following compatibility matrix outlines the support and limitations when mapping data between these two data systems. Please refer to [Amazon Redshift Data Types](https://docs.aws.amazon.com/redshift/latest/dg/c_Supported_data_types.html) and [Apache Iceberg Table Specifications](https://iceberg.apache.org/spec/#primitive-types) for more details on supported data types in respective data systems. - -Redshift data type | Aliases | Iceberg data type ----|---|--- -SMALLINT | INT2 | int -INTEGER | INT, INT4 | int -BIGINT | INT8 | long -DECIMAL | NUMERIC | decimal -REAL | FLOAT4 | float -REAL | FLOAT4 | float -DOUBLE PRECISION | FLOAT8, FLOAT | double -CHAR | CHARACTER, NCHAR | string -VARCHAR | CHARACTER VARYING, NVARCHAR | string -BPCHAR | | string -TEXT | | string -DATE | | date -TIME | TIME WITHOUT TIMEZONE | time -TIME | TIME WITH TIMEZONE | not supported -TIMESTAMP | TIMESTAMP WITHOUT TIMEZONE | TIMESTAMP -TIMESTAMPZ | TIMESTAMP WITH TIMEZONE | TIMESTAMPZ -INTERVAL YEAR TO MONTH | | Not supported -INTERVAL DAY TO SECOND | | Not supported -BOOLEAN | BOOL | bool -HLLSKETCH | | Not supported -SUPER | | Not supported -VARBYTE | VARBINARY, BINARY VARYING | binary -GEOMETRY | | Not supported -GEOGRAPHY | | Not supported - -## Considerations and limitations when using AWS Glue Iceberg REST Catalog APIs - -Following are the considerations and limitations when using the Apache Iceberg REST Catalog Data Definition Language (DDL) operation behavior. - -###### Considerations - - * **DeleteTable API behavior** – DeleteTable API supports a purge option. When purge is set to `true`, the table data is deleted, otherwise data is not deleted. For tables in Amazon S3, the operation will not delete table data. The operation fails when table is stored in Amazon S3 and `purge = TRUE,` . - -For tables stored in Amazon Redshift managed storage, the operation will delete table data, similar to `DROP TABLE`behavior in Amazon Redshift. The operation fails when table is stored in Amazon Redshift and `purge = FALSE`. - - * **CreateTable API behavior** – The `CreateTable` API operation doesn't support the option `state-create = TRUE`. - - * **`RenameTable` API behavior** – The `RenameTable` operation is supported in tables in Amazon Redshift but not in Amazon S3. - - * **DDL operations for namespaces and tables in Amazon Redshift** – Create, Update, Delete operations for namespaces and tables in Amazon Redshift are asynchronous operations because they are dependent on when Amazon Redshift managed workgroup is available and whether a conflicting DDL and DML transaction is in progress and operation has to wait for lock and then attempt to commit changes. - -During a create, update or delete operation, the endpoint returns a 202 response with the following payload. - - {