AWS prescriptive-guidance documentation change
Summary
Restructured migration documentation with removed technical implementation details, added section links, and simplified strategy comparison. Removed specific EMR/Hive metastore workaround instructions and migration procedure comparisons.
Security assessment
Changes focus on documentation structure and migration strategy guidance without addressing security vulnerabilities or introducing security features. Removed content about EMR cluster configurations but doesn't indicate security risks.
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md index b06abb30d..ff2900791 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/table-migration.md @@ -5 +5 @@ -In-place migrationFull data migrationChoosing a migration strategy +# Migrating existing tables to Iceberg @@ -7 +7 @@ In-place migrationFull data migrationChoosing a migration strategy -# Migrating existing tables to Apache Iceberg +This section focuses on migrating your existing Hive-style tables to Iceberg format. It applies to tables that use traditional Hive-compatible formats such as [Apache Parquet](https://parquet.apache.org/) or [Apache ORC](https://orc.apache.org/). This information doesn't apply to tables that already use modern table formats such as Linux Foundation Delta Lake or Apache Hudi. @@ -9 +9 @@ In-place migrationFull data migrationChoosing a migration strategy -To migrate your current Athena or AWS Glue tables (also known as Hive tables) to Iceberg format, you can use either in-place or full data migration: +To migrate your current Hive-style tables to Iceberg format, you can use either in-place or full data migration: @@ -11 +11 @@ To migrate your current Athena or AWS Glue tables (also known as Hive tables) to - * **In-place migration** is the process of generating Iceberg's metadata files on top of existing data files. + * [In-place migration](./table-migration-inplace.html) is the process of generating Iceberg's metadata files on top of existing data files. @@ -13 +13 @@ To migrate your current Athena or AWS Glue tables (also known as Hive tables) to - * **Full data migration** creates the Iceberg metadata layer and also rewrites existing data files from the original table to the new Iceberg table. + * [Full data migration](./table-migration-full.html) creates the Iceberg metadata layer and also rewrites existing data files from the original table to the new Iceberg table. @@ -18 +18 @@ To migrate your current Athena or AWS Glue tables (also known as Hive tables) to -The following sections provide an overview of the APIs available to migrate tables and guidance for choosing a migration strategy. For more information about these two strategies, see the [Table Migration](https://iceberg.apache.org/docs/latest/table-migration/) section in the Iceberg documentation. +The following sections provide a detailed overview of each migration method, including step-by-step instructions and considerations for implementation. For more information about these migration strategies, see the [Table Migration](https://iceberg.apache.org/docs/latest/table-migration/) section of the Iceberg documentation. @@ -20 +20 @@ The following sections provide an overview of the APIs available to migrate tabl -## In-place migration +After you review the details of the in-place and full data migration methods, see the following two key sections to aid your decision-making process: @@ -22 +22 @@ The following sections provide an overview of the APIs available to migrate tabl -In-place migration eliminates the need to rewrite all the data files. Instead, Iceberg metadata files are generated and linked to your existing data files. Iceberg offers three options for implementing in-place migration: + * [Choosing a migration strategy](./migration-strategy.html) provides guidance through a series of questions and scenarios, to help you determine the most suitable migration approach based on your specific requirements and use cases. @@ -24 +24 @@ In-place migration eliminates the need to rewrite all the data files. Instead, I - * Using the `snapshot` procedure, as explained in the sections [Snapshot Table](https://iceberg.apache.org/docs/latest/spark-procedures/#snapshot) and [Spark procedure: snapshot](https://iceberg.apache.org/docs/latest/spark-procedures/#snapshot) in the Iceberg documentation. + * [Migration options summary](./migration-options.html) provides a comprehensive table that compares key characteristics and considerations across different migration options. This table serves as a quick reference guide and offers a feature comparison to help you understand the technical trade-offs between methods. @@ -26,134 +25,0 @@ In-place migration eliminates the need to rewrite all the data files. Instead, I - * Using the `add_files` procedure, as explained in the sections [Add Files](https://iceberg.apache.org/docs/latest/table-migration/#add-files) and [Spark procedure: add_files](https://iceberg.apache.org/docs/latest/spark-procedures/#add_files) in the Iceberg documentation. - - * Using the `migrate` procedure, as explained in the sections [Migrate Table](https://iceberg.apache.org/docs/latest/table-migration/#migrate-table) and [Spark procedure: Migrate](https://iceberg.apache.org/docs/latest/spark-procedures/#migrate) in the Iceberg documentation. - - - - -Currently, the migrate procedure doesn't work directly with the AWS Glue Data Catalog—it works only with the Hive metastore. If you have a requirement to use the `migrate` procedure instead of `snapshot` or `add_files`, you can use a temporary Amazon EMR cluster with the Hive metastore (HMS). This approach requires Iceberg version 1.2 or later. - -Let's say you want to create the following Hive table: - - - -You can create this Hive table by running this code in the Athena console: - - - CREATE EXTERNAL TABLE 'hive_table'( - 'id' bigint, - 'data' string) - USING parquet - LOCATION - 's3://datalake-xxxx/aws_workshop/iceberg_db/hive_table' - - INSERT INTO iceberg_db.hive_table VALUES (1, 'a') - -If your Hive table is partitioned, include the partition statement and add the partitions according to Hive requirements. - - - ALTER TABLE default.placeholder_table_for_migration ADD - PARTITION (date = '2023-10-10') - -Steps: - - 1. Create an Amazon EMR cluster **without** enabling the AWS Glue Data Catalog integration—that is, don't select the check boxes for Hive or Spark table metadata. That's because you will use the native Hive metastore (HMS) that's available in the cluster for this workaround. - - - - 2. Configure the Spark session to use the Iceberg Hive catalog implementation. - - "spark.sql.extensions":"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", - "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog", - "spark.sql.catalog.spark_catalog.type": "hive", - - 3. Validate that your Amazon EMR cluster isn't connected to the AWS Glue Data Catalog by running `show databases` or `show tables`. - - - - 4. Register the Hive table in the Hive metastore of your Amazon EMR cluster, and then use the Iceberg `migrate` procedure. - - - -This procedure creates the Iceberg metadata files in the same location as the Hive table. - - 5. Register the migrated Iceberg table in the AWS Glue Data Catalog. - - 6. Switch back to an Amazon EMR cluster that has the AWS Glue Data Catalog integration enabled. - - - - 7. Use the following Iceberg configuration in the Spark session. - - "spark.sql.extensions":"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", - "spark.sql.catalog.glue_catalog": "org.apache.iceberg.spark.SparkCatalog", - "spark.sql.catalog.glue_catalog.warehouse": "s3://datalake-xxxx/aws_workshop", - "spark.sql.catalog.glue_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", - "spark.sql.catalog.glue_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO" - - - - -You can now query this table from Amazon EMR, AWS Glue, or Athena. - - - -## Full data migration - -Full data migration recreates the data files as well as the metadata. This approach takes longer and requires additional computing resources compared with in-place migration. However, this option helps improve table quality: You can validate the data, make schema and partition changes, resort the data, and so on. To implement full data migration, use one of the following options: - - * Use the `CREATE TABLE ... AS SELECT` ([CTAS](https://iceberg.apache.org/docs/latest/spark-ddl/#create-table--as-select)) statement in Spark on Amazon EMR, AWS Glue, or Athena. You can set the partition specification and table properties for the new Iceberg table by using the `PARTITIONED BY` and `TBLPROPERTIES` clauses. You can fine-tune the schema and partitioning for the new table according to your needs instead of simply inheriting them from the source table. - - * Read from the source table and write the data as a new Iceberg table by using Spark on Amazon EMR or AWS Glue (see [Creating a table](https://iceberg.apache.org/docs/nightly/spark-getting-started/#creating-a-table) in the Iceberg documentation). - - - - -## Choosing a migration strategy - -To choose the best migration strategy, consider the questions in the following table. - -**Question** | **Recommendation** ----|--- -**What is the data file format (for example, CSV or Apache Parquet)?** | - - * Consider in-place migration if your table file format is Parquet, ORC, or Avro. - - - - * For other formats such as CSV, JSON, and so on, use full data migration. - - -**Do you want to update or consolidate the table schema?** | - - * If you want to evolve the table schema by using Iceberg native capabilities, consider in-place migration. For example, you can rename columns after the migration. (The schema can be changed in the Iceberg metadata layer.) - - - - * If you want to delete entire columns from data files, we recommend that you use full data migration. - - -**Would the table benefit from changing the partition strategy?** | - - * If Iceberg's partitioning approach meets your requirements (for example, new data is stored by using the new partition layout while existing partitions remain as is), consider in-place migration. - - - - * If you want to use hidden partitions in your table, consider full data migration. For more information about hidden partitions, see the [Best practices](./best-practices-read.html#read-partitioning-hidden) section. - - -**Would the table benefit from adding or changing the sort order strategy?** | - - * Adding or changing the sort order of your data requires rewriting the dataset. In this case, consider using full data migration. - - - - * For large tables where it's prohibitively expensive to rewrite all the table partitions, consider using in-place migration and run compaction (with sorting enabled) for the most frequently accessed partitions. - - -**Does the table have many small files?** | - - * Merging small files into larger files requires rewriting the dataset. In this case, consider using full data migration. - - - - * For large tables where it's prohibitively expensive to rewrite all the table partitions, consider using in-place migration and run compaction (with sorting enabled) for the most frequently accessed partitions. @@ -169 +35 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Working with Iceberg tables by using Athena SQL +Working with Iceberg tables by using PyIceberg @@ -171 +37 @@ Working with Iceberg tables by using Athena SQL -Best practices for optimizing Iceberg workloads +In-place migration