AWS prescriptive-guidance documentation change
Summary
Completely restructured and expanded the Apache Iceberg table migration documentation. Added detailed sections on in-place migration (snapshot/migrate procedures), full data migration, migration strategy selection, and comprehensive comparison tables. Included implementation examples, AWS Glue Data Catalog workarounds, and synchronization techniques.
Security assessment
The changes focus entirely on migration procedures, data management, and performance optimization. There are no references to security vulnerabilities, access controls, encryption, authentication, or security incidents. The content discusses metadata handling and data synchronization without security implications.
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md index 29cfda609..1c45240dd 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md @@ -6,0 +7,2 @@ +In-place migrationFull data migrationChoosing a migration strategyMigration options summary + @@ -13 +15 @@ To migrate your current Hive-style tables to Iceberg format, you can use either - * [In-place migration](./table-migration-inplace.html) is the process of generating Iceberg's metadata files on top of existing data files. + * **In-place migration** is the process of generating Iceberg's metadata files on top of existing data files. @@ -15 +17 @@ To migrate your current Hive-style tables to Iceberg format, you can use either - * [Full data migration](./table-migration-full.html) creates the Iceberg metadata layer and also rewrites existing data files from the original table to the new Iceberg table. + * **Full data migration** creates the Iceberg metadata layer and also rewrites existing data files from the original table to the new Iceberg table. @@ -24 +26,461 @@ After you review the details of the in-place and full data migration methods, se - * [Choosing a migration strategy](./migration-strategy.html) provides guidance through a series of questions and scenarios, to help you determine the most suitable migration approach based on your specific requirements and use cases. + * Choosing a migration strategy provides guidance through a series of questions and scenarios, to help you determine the most suitable migration approach based on your specific requirements and use cases. + + * Migration options summary provides a comprehensive table that compares key characteristics and considerations across different migration options. This table serves as a quick reference guide and offers a feature comparison to help you understand the technical trade-offs between methods. + + + + +## In-place migration + +In-place migration eliminates the need to rewrite all your data files. Instead, Iceberg metadata files are generated and linked to your existing data files. This method is typically faster and more cost-effective, especially for large datasets or tables that have compatible file formats such as Parquet, Avro, and ORC. + +###### Note + +In-place migration cannot be used when migrating to [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html). + +Iceberg offers two main options for implementing in-place migration: + + * Using the [snapshot](https://iceberg.apache.org/docs/latest/spark-procedures/#snapshot) procedure to create a new Iceberg table while keeping the source table unchanged. For more information, see [Snapshot Table](https://iceberg.apache.org/docs/latest/table-migration/#snapshot-table) in the Iceberg documentation. + + * Using the [migrate](https://iceberg.apache.org/docs/latest/spark-procedures/#migrate) procedure to create a new Iceberg table as a substitution for the source table. For more information, see [Migrate Table](https://iceberg.apache.org/docs/latest/table-migration/#migrate-table) in the Iceberg documentation. Although this procedure works with Hive Metastore (HMS), it isn't currently compatible with the AWS Glue Data Catalog. The _Replicating the table migration procedure in AWS Glue Data Catalog_ section later in this guide provides a workaround for achieving a similar outcome with the Data Catalog. + + + + +After you perform in-place migration by using either `snapshot` or `migrate`, some data files might remain unmigrated. This typically happens when writers continue writing to the source table during or after migration. To incorporate these remaining files into your Iceberg table, you can use the [add_files](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) procedure. For more information, see [Add Files](https://iceberg.apache.org/docs/latest/table-migration/#add-files) in the Iceberg documentation. + +Let's say you have a Parquet-based `products` table that was created and populated in Athena as follows: + + + CREATE EXTERNAL TABLE mydb.products ( + product_id INT, + product_name STRING + ) + PARTITIONED BY (category STRING) + STORED AS PARQUET + LOCATION 's3://DOC-EXAMPLE-BUCKET/products/'; + + INSERT INTO mydb.products + VALUES + (1001, 'Smartphone', 'electronics'), + (1002, 'Laptop', 'electronics'), + (2001, 'T-Shirt', 'clothing'), + (2002, 'Jeans', 'clothing'); + +The following sections explain how you can use the `snapshot` and `migrate` procedures with this table. + +### Option 1: snapshot procedure + +The `snapshot` procedure creates a new Iceberg table that has a different name but replicates the schema and partitioning of the source table. This operation leaves the source table completely unchanged both during and after the action. It effectively creates a lightweight copy of the table, which is particularly useful for testing scenarios or data exploration without risking modifications to the original data source. This approach enables a transition period where both the original table and the Iceberg table remain available (see the notes at the end of this section). When testing is complete, you can move your new Iceberg table to production by transitioning all writers and readers to the new table. + +You can run the `snapshot` procedure by using Spark in any Amazon EMR deployment model (for example, Amazon EMR on EC2, Amazon EMR on EKS, EMR Serverless) and AWS Glue. + +To test in-place migration with the `snapshot` Spark procedure, follow these steps: + + 1. Launch a Spark application and configure the Spark session with the following settings: + + * `"spark.sql.extensions":"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"` + + * `"spark.sql.catalog.spark_catalog":"org.apache.iceberg.spark.SparkSessionCatalog"` + + * `"spark.sql.catalog.spark_catalog.type":"glue"` + + * `"spark.hadoop.hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"` + + 2. Run the `snapshot` procedure to create a new Iceberg table that points to the original table data files: + + spark.sql(f""" + CALL system.snapshot( + source_table => 'mydb.products', + table => 'mydb.products_iceberg', + location => 's3://DOC-EXAMPLE-BUCKET/products_iceberg/' + ) + """ + ).show(truncate=False) + +The output dataframe contains the `imported_files_count` (the numbers of files that were added). + + 3. Validate the new table by querying it: + + spark.sql(f""" + SELECT * FROM mydb.products_iceberg LIMIT 10 + """ + ).show(truncate=False) + + + + +Notes: + + * After you run the procedure, any data file modifications on the source table will throw the generated table out of sync. New files that you add won't be visible in the Iceberg table, and files that you removed will affect query capabilities in the Iceberg table. To avoid the synchronization issues: + + * If the new Iceberg table is intended for production use, stop all processes that write to the original table and redirect them to the new table. + + * If you need a transition period or if the new Iceberg table is for testing purposes, see Keeping Iceberg tables in sync after in-place migration later in this section for guidance on maintaining table synchronization. + + * When you use the `snapshot` procedure, the `gc.enabled` property is set to `false` in the table properties of the created Iceberg table. This setting prohibits actions such as `expire_snapshots`, `remove_orphan_files`, or `DROP TABLE` with the `PURGE` option, which would physically delete data files. Iceberg delete or merge operations, which do not directly impact source files, are still allowed. + + * To make your new Iceberg table fully functional, with no limits on actions that physically delete data files, you can change the `gc.enabled` table property to `true`. However, this setting will allow actions that impact source data files, which could corrupt access to the original table. Therefore, change the `gc.enabled` property only if you no longer need to maintain the original table's functionality. For example: + + spark.sql(f""" + ALTER TABLE mydb.products_iceberg + SET TBLPROPERTIES ('gc.enabled' = 'true'); + """) + + + + +### Option 2: migrate procedure + +The `migrate` procedure creates a new Iceberg table that has the same name, schema, and partitioning as the source table. When this procedure runs, it locks the source table and renames it to `<table_name>_BACKUP_` (or a custom name specified by the `backup_table_name` procedure parameter). + +###### Note + +If you set the `drop_backup` procedure parameter to `true`, the original table will not be retained as a backup. + +Consequently, the `migrate` table procedure requires all modifications that affect the source table to be stopped before the action is performed. Before you run the `migrate` procedure: + + * Stop all writers that interact with the source table. + + * Modify readers and writers that don't natively support Iceberg to enable Iceberg support. + + + + +For example: + + * Athena continues to work without modification. + + * Spark requires: + + * Iceberg Java Archive (JAR) files to be included in the classpath (see the [Working with Iceberg in Amazon EMR](./iceberg-emr.html) and [Working with Iceberg in AWS Glue](./iceberg-glue.html) sections earlier in this guide). + + * The following Spark session catalog configurations (using `SparkSessionCatalog` to add Iceberg support while maintaining built-in catalog functionalities for non-Iceberg tables): + + * `"spark.sql.extensions":"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"` + + * `"spark.sql.catalog.spark_catalog":"org.apache.iceberg.spark.SparkSessionCatalog"` + + * `"spark.sql.catalog.spark_catalog.type":"glue"` + + * `"spark.hadoop.hive.metastore.client.factory.class":"com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"` + + + + +After you run the procedure, you can restart your writers with their new Iceberg configuration. + +Currently, the `migrate` procedure isn't compatible with the AWS Glue Data Catalog, because the Data Catalog doesn't support the `RENAME` operation. Therefore, we recommend that you use this procedure only when you're working with Hive Metastore. If you're using the Data Catalog, see the next section for an alternative approach. + +You can run the `migrate` procedure across all Amazon EMR deployment models (Amazon EMR on EC2, Amazon EMR on EKS, EMR Serverless) and AWS Glue, but it requires a configured connection to Hive Metastore. Amazon EMR on EC2 is the recommended choice because it provides a built-in Hive Metastore configuration, which minimizes setup complexity. + +To test in-place migration with the `migrate` Spark procedure from an Amazon EMR on EC2 cluster that's configured with Hive Metastore, follow these steps: + + 1. Launch a Spark application and configure the Spark session to use the Iceberg Hive catalog implementation. For example, if you're using the `pyspark` CLI: + + pyspark --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=hive + + 2. Create a `products` table in Hive Metastore. This is the source table, which already exists in a typical migration. + + 1. Create the `products` external Hive table in Hive Metastore to point to the existing data in Amazon S3: + + spark.sql(f""" + CREATE EXTERNAL TABLE products ( + product_id INT, + product_name STRING + ) + PARTITIONED BY (category STRING) + STORED AS PARQUET + LOCATION 's3://DOC-EXAMPLE-BUCKET/products/'; + """ + ) + + 2. Add the existing partitions by using the `MSCK REPAIR TABLE` command: + + spark.sql(f""" + MSCK REPAIR TABLE products + """ + ) + + 3. Confirm that the table contains data by running a `SELECT` query: + + spark.sql(f""" + SELECT * FROM products + """ + ).show(truncate=False)