AWS prescriptive-guidance documentation change
Summary
Updated documentation to reflect AWS Glue 5.0 support, restructured custom Iceberg version implementation instructions, and simplified Spark configuration examples
Security assessment
The changes primarily focus on version updates (adding Glue 5.0), simplifying configuration processes, and updating implementation patterns for custom Iceberg versions. There is no mention of security vulnerabilities, patches, or explicit security enhancements. The modifications emphasize version control and compatibility rather than addressing security concerns.
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-glue.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-glue.md index b79924467..bd4f28edd 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-glue.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/iceberg-glue.md @@ -17 +17 @@ When you create Iceberg jobs in AWS Glue, depending on the version of AWS Glue, -AWS Glue versions 3.0 and 4.0 natively support transactional data lake formats such as Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake in AWS Glue for Spark. This integration feature simplifies the configuration steps required to start using these frameworks in AWS Glue. +AWS Glue versions 3.0, 4.0, and 5.0 natively support transactional data lake formats such as Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake in AWS Glue for Spark. This integration feature simplifies the configuration steps required to start using these frameworks in AWS Glue. @@ -30,47 +30 @@ If you are authoring a job by using a notebook, you can configure the parameter -## Using a custom Iceberg version - -In some situations, you might want to retain control over the Iceberg version for the job and upgrade it at your own pace. For example, upgrading to a later version can unlock access to new features and performance enhancements. To use a specific Iceberg version with AWS Glue, you can use a custom connector or your own JAR file. - -### Using a custom connector - -AWS Glue supports connectors, which are optional code packages that assist with accessing data stores in AWS Glue Studio. You can [subscribe to a connector](https://docs.aws.amazon.com/glue/latest/dg/connectors-chapter.html#subscribe-marketplace-connectors) in AWS Marketplace, or you can create a custom connector. - -###### Note - -AWS Marketplace offers the [Apache Iceberg connector for AWS Glue](https://aws.amazon.com/marketplace/pp/prodview-iicxofvpqvsio). However, we recommend that you use a custom connector instead to retain control over Iceberg versions. - -For example, to create a customer connector for Iceberg version 0.13.1, follow these steps: - - 1. Upload the files `iceberg-spark-runtime-3.1_2.12-0.13.1.jar`, `bundle-2.17.161.jar`, and `url-connection-client-2.17.161.jar` to an Amazon S3 bucket. You can download these files from their respective Apache Maven repositories. - - 2. On the [AWS Glue Studio console](https://console.aws.amazon.com/gluestudio/), create a custom Spark connector: - - 1. In the navigation pane, choose **Data connections**. (If you're using the older navigation, choose **Connectors** , **Create custom connector**.) - - 2. In the **Connectors** box, choose **Create custom connector**. - - 3. On the **Create custom connector** page: - - * Specify the path to the JAR files in Amazon S3. - - * Enter a name for the connector. - - * Choose **Spark** as the connector type. - - * For **Class name** , specify the fully qualified data source class name (or its alias) that you use when loading the Spark data source with the `format` operator. - - * (Optional) Provide a description of the connector. - - - - -3\. Choose **Create connector**. - -When you work with connectors in AWS Glue, you must create a connection for the connector. A connection contains the properties that are required to connect to a particular data store. You use the connection with your data sources and data targets in the ETL job. Connectors and connections work together to facilitate access to the data stores. - -To create a connection by using the custom Iceberg connector you created: - - 1. On the [AWS Glue Studio console](https://console.aws.amazon.com/gluestudio/), select your custom Iceberg connector. - - 2. Follow the prompts to supply the details, such as your VPC and other network configurations required by the job, and then choose **Create connection**. - +The `iceberg` configuration for `--datalake-formats` in AWS Glue corresponds to specific Iceberg versions based on your AWS Glue version: @@ -77,0 +32,5 @@ To create a connection by using the custom Iceberg connector you created: +AWS Glue version | Default Iceberg version +---|--- +5.0 | 1.7.1 +4.0 | 1.0.0 +3.0 | 0.13.1 @@ -78,0 +38 @@ To create a connection by using the custom Iceberg connector you created: +## Using a custom Iceberg version @@ -80 +40 @@ To create a connection by using the custom Iceberg connector you created: -You can now use the connection in your AWS Glue ETL job. Depending on how you create the job, there are different ways to attach the connection to your job: +In some situations, you might want to retain control over the Iceberg version for the job and upgrade it at your own pace. For example, upgrading to a later version can unlock access to new features and performance enhancements. To use a specific Iceberg version with AWS Glue, you can provide your own JAR files. @@ -82 +42 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - * If you create a visual job by using AWS Glue Studio, you can select the connection from the **Connection** list on the **Data source properties – Connector** tab. +Before you implement a custom Iceberg version, verify compatibility with your AWS Glue environment by checking the [AWS Glue versions](https://docs.aws.amazon.com/glue/latest/dg/release-notes.html) section of the AWS Glue documentation. For example, AWS Glue 5.0 requires compatibility with Spark 3.5.4. @@ -84 +44 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - * If you develop the job in a notebook, use the `%connections` magic to set the connection name: +As an example, to run AWS Glue jobs that use Iceberg version 1.9.1, follow these steps: @@ -86 +46 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - %glue_version 3.0 + 1. Acquire and upload the required JAR files to Amazon S3: @@ -88 +48 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - %connections <name-of-the iceberg-connection> + 1. Download [iceberg-spark-runtime-3.5_2.12-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-spark-runtime-3.5_2.12/1.9.1) and [iceberg-aws-bundle-1.9.1.jar](https://mvnrepository.com/artifact/org.apache.iceberg/iceberg-aws-bundle/1.9.1) from the Apache Maven repository. @@ -90,5 +50 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - %%configure - { - "--conf" : "job-specific Spark configurations, to be discussed later", - "--datalake-formats" : "iceberg" - } + 2. Upload these files to your designated S3 bucket location (for example, `s3://your-bucket-name/jars/`). @@ -96 +52 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr - * If you author the job by using the script editor, specify the connection on the **Job details** tab, under **Advanced properties** , **Additional network connections**. + 2. Set up the job parameters for your AWS Glue job as follows: @@ -97,0 +54 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr + 1. Specify the complete S3 path to both JAR files in the `--extra-jars` parameter, separating them with a comma (for example, `s3://your-bucket-name/jars/iceberg-spark-runtime-3.5_2.12-1.9.1.jar,s3://your-bucket-name/jars/iceberg-aws-bundle-1.9.1.jar`). @@ -98,0 +56 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr + 2. Do not include `iceberg` as a value for the `--datalake-formats` parameter. @@ -99,0 +58 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr + 3. If you use AWS Glue 5.0, you must set the `--user-jars-first` parameter to `true`. @@ -101 +59,0 @@ You can now use the connection in your AWS Glue ETL job. Depending on how you cr -For more information about the procedures in this section, see [Using connectors and connections with AWS Glue Studio](https://docs.aws.amazon.com/glue/latest/dg/connectors-chapter.html) in the AWS Glue documentation. @@ -103 +60,0 @@ For more information about the procedures in this section, see [Using connectors -### Bringing your own JAR files @@ -105 +61,0 @@ For more information about the procedures in this section, see [Using connectors -In AWS Glue, you can also work with Iceberg without having to use a connector. This approach is useful when you want to retain control over the Iceberg version and quickly update it. To use this option, upload the required Iceberg JAR files into an S3 bucket of your choice and reference the files in your AWS Glue job. For example, if you're working with Iceberg 1.0.0, the required JAR files are `iceberg-spark-runtime-3.0_2.12-1.0.0.jar`, `url-connection-client-2.15.40.jar`, and `bundle-2.15.40.jar`. You can also prioritize the additional JAR files in the class path by setting the `--user-jars-first` parameter to `true` for the job. @@ -112,3 +68 @@ This section discusses the Spark configurations required to author an AWS Glue E - %glue_version 3.0 - - %connections <name-of-the iceberg-connection> + %glue_version 5.0 @@ -124 +78 @@ Configure the Spark session with the following properties: - * `<catalog_name>` is your Iceberg Spark session catalog name. Replace it with the name of your catalog, and remember to change the references throughout all configurations that are associated with this catalog. In your code, you should then refer to your Iceberg tables with the fully qualified table name, including the Spark session catalog name, as follows: `<catalog_name>.<database_name>.<table_name>`. + * `<catalog_name>` is the name of your Iceberg Spark session catalog name. Replace it with a name of your choice, and remember to change the references throughout all configurations that are associated with this catalog. In your code, you can refer to your Iceberg tables by using the fully qualified table name, including the Spark session catalog name, as follows: @@ -126 +80,3 @@ Configure the Spark session with the following properties: - * `<catalog_name>.<warehouse>` points to the Amazon S3 path where you want to store your data and metadata. +`<catalog_name>.<database_name>.<table_name>` + +Alternatively, you can change the default catalog to the Iceberg catalog that you defined by setting `spark.sql.defaultCatalog` to your catalog name. You can use this second approach to refer to tables without the catalog prefix, which can simplify your queries. @@ -128 +84 @@ Configure the Spark session with the following properties: - * To make the catalog an AWS Glue Data Catalog, set `<catalog_name>.catalog-impl` to `org.apache.iceberg.aws.glue.GlueCatalog`. This key is required to point to an implementation class for any custom catalog implementation. For catalogs supported by Iceberg, see the [General best practices](./best-practices.html)[General best practices](./best-practices-general.html) section later in this guide. + * `<catalog_name>.<warehouse>` points to the Amazon S3 path where you want to store your data and metadata. @@ -130 +86 @@ Configure the Spark session with the following properties: - * Use `org.apache.iceberg.aws.s3.S3FileIO` as the `<catalog_name>.io-impl` in order to take advantage of Amazon S3 multipart upload for high parallelism. + * To make the catalog an AWS Glue Data Catalog, set `spark.sql.catalog.<catalog_name>.type` to `glue`. This key is required to point to an implementation class for any custom catalog implementation. For catalogs supported by Iceberg, see the [General best practices](./best-practices.html) section later in this guide. @@ -140,6 +96,2 @@ For example, if you have a catalog called `glue_iceberg`, you can configure your - "‐‐datalake-formats" : "iceberg", - "‐‐conf" : "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", - "‐‐conf" : "spark.sql.catalog.glue_iceberg=org.apache.iceberg.spark.SparkCatalog", - "‐‐conf" : "spark.sql.catalog.glue_iceberg.warehouse=s3://<your-warehouse-dir>=>/", - "‐‐conf" : " spark.sql.catalog.glue_iceberg.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog ", - "‐‐conf" : " spark.sql.catalog.glue_iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO + "--datalake-formats" : "iceberg", + "--conf" : "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.glue_iceberg=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_iceberg.warehouse=s3://<your-warehouse-dir>/ --conf spark.sql.catalog.glue_iceberg.type=glue" @@ -156,2 +107 @@ Alternatively, you can use code to add the above configurations to your Spark sc - .config("spark.sql.catalog.glue_iceberg.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \ - .config("spark.sql.catalog.glue_iceberg.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \ + .config("spark.sql.catalog.glue_iceberg.type", "glue") \ @@ -170 +120 @@ This section provides general guidelines for tuning Spark jobs in AWS Glue to op - * **Use custom connectors or add library dependencies** \- AWS Glue native integration for Iceberg is best for getting started with Iceberg. However, for production workloads, we recommend that you use custom containers or add library dependencies (as discussed earlier in this guide) to get full control over the Iceberg version. This approach helps you benefit from the latest Iceberg features and performance improvements in your AWS Glue jobs. + * **Use the desired Iceberg version** – AWS Glue native integration for Iceberg is best for getting started with Iceberg. However, for production workloads, we recommend that you add library dependencies (as discussed earlier in this guide) to get full control over the Iceberg version. This approach helps you benefit from the latest Iceberg features and performance improvements in your AWS Glue jobs.