AWS prescriptive-guidance documentation change
Summary
Updated documentation about AWS services supporting Apache Iceberg, including expanded descriptions of Amazon S3 Tables integration, SageMaker lakehouse architecture, and enhanced AWS Glue capabilities
Security assessment
Changes focus on feature enhancements, service integrations, and architectural descriptions. While security capabilities of S3 are mentioned in passing, no new security vulnerabilities or explicit security features are introduced in these changes.
Diff
diff --git a/prescriptive-guidance/latest/apache-iceberg-on-aws/data-lakes.md b/prescriptive-guidance/latest/apache-iceberg-on-aws/data-lakes.md index 0f5292518..665f08daa 100644 --- a//prescriptive-guidance/latest/apache-iceberg-on-aws/data-lakes.md +++ b//prescriptive-guidance/latest/apache-iceberg-on-aws/data-lakes.md @@ -10,0 +11,2 @@ Advanced use cases in modern data lakesIntroduction to Apache IcebergAWS support +The evolution of data storage has progressed from databases to data warehouses and data lakes, where each technology addresses unique business and data requirements. Traditional databases excelled at handling structured data and transactional workloads, but they faced performance challenges as data volumes increased. Data warehouses emerged to tackle performance and scalability issues, but like databases, they relied on proprietary formats within vertically integrated systems. + @@ -13 +15 @@ Data lakes offer one of the best options for storing data in terms of cost, scal -Despite these benefits, data lakes weren't initially designed with database-like capabilities. A data lake doesn't provide support for atomicity, consistency, isolation, and durability (ACID) processing semantics, which you might require to effectively optimize and manage your data at scale across hundreds or thousands of users by using a multitude of different technologies. Data lakes don't provide native support for the following functionality: +Despite these benefits, data lakes weren't initially designed with database-like capabilities. A data lake doesn't provide support for atomicity, consistency, isolation, and durability (ACID) processing semantics, which you might require to optimize and manage your data effectively at scale across hundreds or thousands of users by using many different technologies. Data lakes don't provide native support for the following functionality: @@ -32 +34,3 @@ Data lakes that use the traditional Hive-format tables support write operations -To help overcome these challenges, Apache Iceberg provides additional database-like functionality that simplifies the optimization and management overhead of data lakes, while still supporting storage on cost-effective systems such as [Amazon Simple Storage Service (Amazon S3)](https://aws.amazon.com/s3). +These challenges leave users with a dilemma: choose between a fully integrated but proprietary platform, or opt for a vendor-neutral but resource-intensive, self-built data lake that requires constant maintenance and migration to realize its potential value. + +To help overcome these challenges, Iceberg provides additional database-like functionality that simplifies the optimization and management overhead of data lakes, while still supporting storage on cost-effective systems such as [Amazon S3](https://aws.amazon.com/s3). @@ -65 +69 @@ In summary, data lakes that use the Iceberg format benefit from transactional co -Apache Iceberg is supported by popular open source data processing frameworks and by AWS services such as [Amazon EMR](https://aws.amazon.com/pm/emr/), [Amazon Athena](https://aws.amazon.com/athena/), [Amazon Redshift](https://aws.amazon.com/pm/redshift/), and [AWS Glue](https://aws.amazon.com/glue/). The following diagram depicts a simplified reference architecture of a data lake that's based on Iceberg. +Apache Iceberg is supported by AWS services such as [Amazon EMR](https://aws.amazon.com/pm/emr/), [Amazon Athena](https://aws.amazon.com/athena/), [Amazon Redshift](https://aws.amazon.com/pm/redshift/), [AWS Glue](https://aws.amazon.com/glue/), and [Amazon SageMaker](https://aws.amazon.com/sagemaker/). The following diagram depicts a simplified reference architecture of a data lake that's based on Iceberg. @@ -71 +75 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon S3** is the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities. Iceberg was designed and built to interact with Amazon S3 seamlessly, and provides support for many Amazon S3 features as listed in the [Iceberg documentation](https://iceberg.apache.org/docs/latest/aws/#s3-fileio). + * [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) is the best place to build data lakes because of its durability, availability, scalability, security, compliance, and audit capabilities. Iceberg was designed and built to interact with Amazon S3 seamlessly, and provides support for many Amazon S3 features as listed in the [Iceberg documentation](https://iceberg.apache.org/docs/latest/aws/#s3-fileio). In addition, [Amazon S3 Tables](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html) deliver the first cloud object store with built-in Iceberg support and streamline storing tabular data at scale. With S3 Tables support for Iceberg, you can easily query your tabular data by using popular AWS and third-party query engines. @@ -73 +77 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon EMR** is a big data solution for petabyte-scale data processing, interactive analytics, and machine learning by using open source frameworks such as Apache Spark, Flink, Trino, and Hive. Amazon EMR can run on customized Amazon Elastic Compute Cloud (Amazon EC2) clusters, Amazon Elastic Kubernetes Service (Amazon EKS), AWS Outposts, or Amazon EMR Serverless. + * [The next generation of SageMaker](https://aws.amazon.com/sagemaker/) is built on an open lakehouse architecture that unifies data access across Amazon S3 data lakes, Amazon Redshift data warehouses, and third-party and federated data sources. These capabilities help you build powerful analytics and AI/ML applications on a single copy of data. The lakehouse is fully compatible with Iceberg, so you have the flexibility to access and query data in place by using the Iceberg REST API. @@ -75 +79 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon Athena** is a serverless, interactive analytics service that's built on open source frameworks. It supports open-table and file formats and provides a simplified, flexible way to analyze petabytes of data where it lives. Athena provides native support for read, time travel, write, and DDL queries for Iceberg and uses the AWS Glue Data Catalog for the Iceberg metastore. + * [Amazon EMR](https://docs.aws.amazon.com/emr/) is a big data solution for petabyte-scale data processing, interactive analytics, and machine learning by using open source frameworks such as Apache Spark, Flink, Trino, and Hive. Amazon EMR can run on customized Amazon Elastic Compute Cloud (Amazon EC2) clusters, Amazon Elastic Kubernetes Service (Amazon EKS), AWS Outposts, or Amazon EMR Serverless. @@ -77 +81 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon Redshift** is a petabyte-scale cloud data warehouse that supports both cluster-based and serverless deployment options. Amazon Redshift Spectrum can query external tables that are registered with the AWS Glue Data Catalog and stored on Amazon S3. Redshift Spectrum also provides support for the Iceberg storage format. + * [Amazon Athena](https://docs.aws.amazon.com/athena/latest/ug/what-is.html) is a serverless, interactive analytics service that's built on open source frameworks. It supports open-table and file formats and provides a simplified, flexible way to analyze petabytes of data where it lives. Athena provides native support for read, time travel, write, and DDL queries for Iceberg and uses the AWS Glue Data Catalog for the Iceberg metastore. @@ -79 +83 @@ The following AWS services provide native Iceberg integrations. There are additi - * **AWS Glue** is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. AWS Glue 3.0 and later versions support the Iceberg framework for data lakes. You can use AWS Glue to perform read and write operations on Iceberg tables in Amazon S3, or work with Iceberg tables by using the AWS Glue Data Catalog. Additional operations such as insert, update, Spark queries, and Spark writes are also supported. + * [Amazon Redshift](https://docs.aws.amazon.com/redshift/) is a petabyte-scale cloud data warehouse that supports both cluster-based and serverless deployment options. Amazon Redshift Spectrum can query external tables that are registered with the AWS Glue Data Catalog and stored on Amazon S3. Redshift Spectrum also provides support for the Iceberg storage format. @@ -81 +85 @@ The following AWS services provide native Iceberg integrations. There are additi - * **AWS Glue Data Catalog** provides a Hive metastore-compatible data catalog service that supports Iceberg tables. + * [AWS Glue](https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html) is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. It is fully integrated with Iceberg. Specifically, you can perform read and write operations on Iceberg tables by using AWS Glue jobs, manage tables through the [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/start-data-catalog.html) (Hive metastore-compatible), discover and register tables automatically by using AWS Glue crawlers, and evaluate data quality in Iceberg tables through the AWS Glue Data Quality feature. The AWS Glue Data Catalog also supports collecting column statistics, calculating and updating the number of distinct values (NDVs) for each column in Iceberg tables, and automatic table optimizations (compaction, snapshot retention, orphan file deletion). AWS Glue also supports zero-ETL integrations from a list of AWS services and third-party applications into Iceberg tables. @@ -83 +87 @@ The following AWS services provide native Iceberg integrations. There are additi - * **AWS Glue crawler** provides automations to register Iceberg tables in the AWS Glue Data Catalog. + * [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Amazon OpenSearch Serverless, Splunk, Apache Iceberg tables, and any custom HTTP or HTTP endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Coralogix, and Elastic. With Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Firehose to transform your data before delivering it. @@ -85 +89 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon Data Firehose** is a fully managed service for delivering real-time streaming data to destinations such as Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Amazon OpenSearch Serverless, Splunk, Apache Iceberg tables, and any custom HTTP or HTTP endpoints owned by supported third-party service providers, including Datadog, Dynatrace, LogicMonitor, MongoDB, New Relic, Coralogix, and Elastic. With Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Firehose to transform your data before delivering it. + * [Amazon Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/what-is.html) is a fully managed Amazon service that lets you use an Apache Flink application to process streaming data. It supports both reading from and writing to Iceberg tables, and enables real-time data processing and analytics. @@ -87 +91 @@ The following AWS services provide native Iceberg integrations. There are additi - * **Amazon SageMaker AI** supports the storage of feature sets in Amazon SageMaker AI Feature Store by using Iceberg format. + * [Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html) supports the storage of feature sets in Amazon SageMaker AI Feature Store by using Iceberg format. @@ -89 +93 @@ The following AWS services provide native Iceberg integrations. There are additi - * **AWS Lake Formation** provides coarse and fine-grained access control permissions to access data, including Iceberg tables consumed by Athena or Amazon Redshift. To learn more about permissions support for Iceberg tables, see the [Lake Formation documentation](https://docs.aws.amazon.com/lake-formation/latest/dg/working-with-services.html). + * [AWS Lake Formation](https://docs.aws.amazon.com/lake-formation/latest/dg/what-is-lake-formation.html) provides coarse and fine-grained access control permissions to access data, including Iceberg tables consumed by Athena or Amazon Redshift. To learn more about permissions support for Iceberg tables, see the [Lake Formation documentation](https://docs.aws.amazon.com/lake-formation/latest/dg/working-with-services.html). @@ -94 +98 @@ The following AWS services provide native Iceberg integrations. There are additi -AWS has a wide range of services that support Iceberg, but covering all these services is beyond the scope of this guide. The following sections cover Spark (batch and structured streaming) on Amazon EMR and AWS Glue, as well as Amazon Athena SQL. The following section provides a quick look at Iceberg support in Athena SQL. +AWS has a wide range of services that support Iceberg, but covering all these services is beyond the scope of this guide. The following sections cover Spark (batch and structured streaming) on Amazon EMR and AWS Glue, as well as Athena SQL. The [following section](./getting-started.html) provides a quick look at Iceberg support in Athena SQL.