AWS AmazonS3 documentation change
Summary
Restructured documentation to add multiple S3 Tables access methods (AWS analytics integration, Iceberg REST endpoint, S3 Tables Catalog), updated prerequisite steps, added configuration examples for Spark sessions/interactive notebooks, and updated client catalog JAR version from 0.1.4 to 0.1.5
Security assessment
The changes add documentation about security configurations including SigV4 signing for REST endpoint access ('spark.sql.catalog.s3_rest_catalog.rest.sigv4-enabled', 'rest.signing-region') and IAM role requirements. While these are security-related features, there is no evidence of addressing a specific vulnerability or security incident.
Diff
diff --git a/AmazonS3/latest/userguide/s3-tables-integrating-glue.md b/AmazonS3/latest/userguide/s3-tables-integrating-glue.md index 96ccd901e..078c00bc0 100644 --- a//AmazonS3/latest/userguide/s3-tables-integrating-glue.md +++ b//AmazonS3/latest/userguide/s3-tables-integrating-glue.md @@ -5 +5 @@ -Prerequisites for S3 Tables AWS Glue ETL jobsCreate an AWS Glue ETL PySpark script to query S3 tablesCreate an AWS Glue ETL job that queries S3 tables +PrerequisitesCreate a script to connect to table bucketsCreate a AWS Glue job that queries tables @@ -11 +11,13 @@ AWS Glue is a serverless data integration service that makes it easy for analyti -An AWS Glue job encapsulates a script that connects to your source data, processes it, and then writes it out to your data target. Typically, a job runs extract, transform, and load (ETL) scripts. Jobs can run scripts designed for Apache Spark and Ray runtime environments or general-purpose Python scripts (Python shell jobs). You can monitor job runs to understand runtime metrics such as completion status, duration, and start time. +An AWS Glue job encapsulates a script that connects to your source data, processes it, and then writes it out to your data target. Typically, a job runs extract, transform, and load (ETL) scripts. Jobs can run scripts designed for Apache Spark runtime environments. You can monitor job runs to understand runtime metrics such as completion status, duration, and start time. + +You can use AWS Glue jobs to process data in your S3 tables by connecting to your tables through the integration with AWS analytics services, or, connect directly using the Amazon S3 Tables Iceberg REST endpoint or the Amazon S3 Tables Catalog for Apache Iceberg. This guide covers the basic steps to get started using AWS Glue with S3 Tables, including: + +###### Topics + + * Prerequisites + + * Create a script to connect to table buckets + + * Create a AWS Glue job that queries tables + + @@ -13 +24,0 @@ An AWS Glue job encapsulates a script that connects to your source data, process -You can use AWS Glue jobs to process data in your S3 tables by connecting to your tables directly using the Amazon S3 Tables Catalog for Apache Iceberg client catalog JAR. @@ -19 +30 @@ S3 Tables is supported on [AWS Glue version 5.0 or higher](https://docs.aws.amaz -## Prerequisites for S3 Tables AWS Glue ETL jobs +## Prerequisites @@ -31 +42,3 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t - * Download the latest version of the Amazon S3 Tables Catalog for Apache Iceberg client catalog JAR from Maven and then upload it to an Amazon S3 bucket. + * (Optional) If you are using the Amazon S3 Tables Catalog for Apache Iceberg you need to download the client catalog JAR and upload it to an S3 bucket. + +###### **Downloading the catalog JAR** @@ -35 +48 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t - wget https://repo1.maven.org/maven2/software/amazon/s3tables/s3-tables-catalog-for-iceberg-runtime/0.1.4/s3-tables-catalog-for-iceberg-runtime-0.1.4.jar + wget https://repo1.maven.org/maven2/software/amazon/s3tables/s3-tables-catalog-for-iceberg-runtime/0.1.5/s3-tables-catalog-for-iceberg-runtime-0.1.5.jar @@ -37 +50 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t - 2. Upload the downloaded JAR to an S3 bucket that your AWS Glue IAM role can access. You can use the following AWS CLI command to upload the JAR. Make sure to replace the `version number` with the latest version, and the `bucket name and path` with your own. + 2. Upload the downloaded JAR to an S3 bucket that your AWS Glue IAM role can access. You can use the following AWS CLI command to upload the JAR. Make sure to replace the `version number` with the latest version, and the `bucket name` and `path` with your own. @@ -39 +52 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t - aws s3 cp s3-tables-catalog-for-iceberg-runtime-0.1.4.jar s3://amzn-s3-demo-bucket1/jars/ + aws s3 cp s3-tables-catalog-for-iceberg-runtime-0.1.5.jar s3://amzn-s3-demo-bucket/jars/ @@ -44 +57 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t -## Create an AWS Glue ETL PySpark script to query S3 tables +## Create a script to connect to table buckets @@ -46 +59 @@ Before you can query tables from a AWS Glue job you must configure an IAM role t -To access your table data when you run an AWS Glue ETL job, you use a PySpark script to configure a Spark session for Apache Iceberg that connects to your S3 table bucket when the job runs. You can modify an existing script to connect to your table buckets or create a new script. For more information on creating AWS Glue scripts, see [Tutorial: Writing an AWS Glue for Spark script](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-intro-tutorial.html) in the _AWS Glue Developer Guide_. +To access your table data when you run an AWS Glue ETL job, you configure a Spark session for Apache Iceberg that connects to your S3 table bucket. You can modify an existing script to connect to your table bucket or create a new script. For more information on creating AWS Glue scripts, see [Tutorial: Writing an AWS Glue for Spark script](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-intro-tutorial.html) in the _AWS Glue Developer Guide_. @@ -48 +61 @@ To access your table data when you run an AWS Glue ETL job, you use a PySpark sc -Use the following code snippet in your PySpark script for configuring Spark's connection to your table bucket. Replace the `placeholder values` with the information for your own table bucket. +You can configure the session to connect to your table buckets through the any of the following S3 Tables access methods: @@ -49,0 +63 @@ Use the following code snippet in your PySpark script for configuring Spark's co + * S3 Tables integration with AWS analytics services @@ -51,8 +65 @@ Use the following code snippet in your PySpark script for configuring Spark's co - # Configure Spark session for Iceberg - spark_conf = SparkSession.builder.appName("GlueJob") - .config("spark.sql.catalog.s3tablesbucket", "org.apache.iceberg.spark.SparkCatalog") - .config("spark.sql.catalog.s3tablesbucket.catalog-impl", "software.amazon.s3tables.iceberg.S3TablesCatalog") - .config("spark.sql.catalog.s3tablesbucket.warehouse", "arn:aws:s3tables:REGION:111122223333:bucket/amzn-s3-demo-table-bucket") - .config("spark.sql.defaultCatalog", "s3tablesbucket") - .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") - .config("spark.sql.catalog.s3tablesbucket.cache-enabled", "false") + * Amazon S3 Tables Iceberg REST endpoint @@ -60 +67 @@ Use the following code snippet in your PySpark script for configuring Spark's co -### Sample script + * Amazon S3 Tables Catalog for Apache Iceberg @@ -62 +68,0 @@ Use the following code snippet in your PySpark script for configuring Spark's co -The following is an sample PySpark script you can use to test querying S3 tables with an AWS Glue job. The script connects to your table bucket and then runs queries to: create a new namespace, create a sample table, insert data into the table, and the table data. To use the script, replace the `placeholder values` with the information for you own table bucket. @@ -65,7 +70,0 @@ The following is an sample PySpark script you can use to test querying S3 tables - import sys - from awsglue.transforms import * - from awsglue.utils import getResolvedOptions - from pyspark.context import SparkContext - from awsglue.context import GlueContext - from awsglue.job import Job - from pyspark.sql import SparkSession @@ -73,8 +72,132 @@ The following is an sample PySpark script you can use to test querying S3 tables - # Configure Spark session for Iceberg - spark_conf = SparkSession.builder.appName("GlueJob") - .config("spark.sql.catalog.s3tablesbucket", "org.apache.iceberg.spark.SparkCatalog") - .config("spark.sql.catalog.s3tablesbucket.catalog-impl", "software.amazon.s3tables.iceberg.S3TablesCatalog") - .config("spark.sql.catalog.s3tablesbucket.warehouse", "arn:aws:s3tables:REGION:111122223333:bucket/amzn-s3-demo-table-bucket") - .config("spark.sql.defaultCatalog", "s3tablesbucket") - .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") - .config("spark.sql.catalog.s3tablesbucket.cache-enabled", "false") +Choose from the following access methods to view setup instructions and configuration examples. + +AWS analytics services integration + + +As a prerequisites to query tables with Spark on AWS Glue using the AWS analytics services integration, you must [Integrate your table buckets with AWS analytics services](./s3-tables-integrating-aws.html) + +You can configure the connection to your table bucket through a Spark session in a job or with AWS Glue Studio magics in an interactive session. To use the following examples, replace the `placeholder values` with the information for your own table bucket. + +**Using a PySpark script** + + +Use the following code snippet in a PySpark script to configure a AWS Glue job to connect to your table bucket using the integration. + + + spark = SparkSession.builder.appName("SparkIcebergSQL") \ + .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ + .config("spark.sql.defaultCatalog","s3tables") + .config("spark.sql.catalog.s3tables", "org.apache.iceberg.spark.SparkCatalog") \ + .config("spark.sql.catalog.s3tables.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog") \ + .config("spark.sql.catalog.s3tables.glue.id", "111122223333:s3tablescatalog/amzn-s3-demo-table-bucket") \ + .config("spark.sql.catalog.s3tables.warehouse", "s3://amzn-s3-demo-table-bucket/warehouse/") \ + .getOrCreate() + +**Using an interactive AWS Glue session** + + +If you are using an interactive notebook session with AWS Glue 5.0, specify the same configurations using the `%%configure` magic in a cell prior to code execution. + + + %%configure + { + "conf": { + "spark.sql.defaultCatalog": "s3tables", + "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + "spark.sql.catalog.s3tables": "org.apache.iceberg.spark.SparkCatalog", + "spark.sql.catalog.s3tables.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", + "spark.sql.catalog.s3tables.glue.id": "111122223333:s3tablescatalog/amzn-s3-demo-table-bucket", + "spark.sql.catalog.s3tables.warehouse": "s3://amzn-s3-demo-table-bucket/warehouse/" + } + } + +Amazon S3 Tables Iceberg REST endpoint + + +You can configure the connection to your table bucket through a Spark session in a job or with AWS Glue Studio magics in an interactive session. To use the following examples, replace the `placeholder values` with the information for your own table bucket. + +**Using a PySpark script** + + +Use the following code snippet in a PySpark script to configure a AWS Glue job to connect to your table bucket using the endpoint. + + + spark = SparkSession.builder.appName("glue-s3-tables-rest") \ + .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ + .config("spark.sql.defaultCatalog", "s3_rest_catalog") \ + .config("spark.sql.catalog.s3_rest_catalog", "org.apache.iceberg.spark.SparkCatalog") \ + .config("spark.sql.catalog.s3_rest_catalog.type", "rest") \ + .config("spark.sql.catalog.s3_rest_catalog.uri", "https://s3tables.Region.amazonaws.com/iceberg") \ + .config("spark.sql.catalog.s3_rest_catalog.warehouse", "arn:aws:s3tables:Region:111122223333:s3tablescatalog/amzn-s3-demo-table-bucket") \ + .config("spark.sql.catalog.s3_rest_catalog.rest.sigv4-enabled", "true") \ + .config("spark.sql.catalog.s3_rest_catalog.rest.signing-name", "s3tables") \ + .config("spark.sql.catalog.s3_rest_catalog.rest.signing-region", "Region") \ + .config('spark.sql.catalog.s3_rest_catalog.io-impl','org.apache.iceberg.aws.s3.S3FileIO') \ + .config('spark.sql.catalog.s3_rest_catalog.rest-metrics-reporting-enabled','false') \ + .getOrCreate() + +**Using an interactive AWS Glue session** + + +If you are using an interactive notebook session with AWS Glue 5.0, specify the same configurations using the `%%configure` magic in a cell prior to code execution. Replace the placeholder values with the information for your own table bucket. + + + %%configure + { + "conf": { + "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", + "spark.sql.defaultCatalog": "s3_rest_catalog", + "spark.sql.catalog.s3_rest_catalog": "org.apache.iceberg.spark.SparkCatalog", + "spark.sql.catalog.s3_rest_catalog.type": "rest", + "spark.sql.catalog.s3_rest_catalog.uri": "https://s3tables.Region.amazonaws.com/iceberg", + "spark.sql.catalog.s3_rest_catalog.warehouse": "arn:aws:s3tables:Region:111122223333:s3tablescatalog/amzn-s3-demo-table-bucket", + "spark.sql.catalog.s3_rest_catalog.rest.sigv4-enabled": "true", + "spark.sql.catalog.s3_rest_catalog.rest.signing-name": "s3tables", + "spark.sql.catalog.s3_rest_catalog.rest.signing-region": "Region", + "spark.sql.catalog.s3_rest_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", + "spark.sql.catalog.s3_rest_catalog.rest-metrics-reporting-enabled": "false" + } + } + +Amazon S3 Tables Catalog for Apache Iceberg + + +As a prerequisite to connecting to tables using the Amazon S3 Tables Catalog for Apache Iceberg you must first download the latest catalog jar and upload it to an S3 bucket. Then, when you create your job, you add the the path to the client catalog JAR as a special parameter. For more information on job parameters in AWS Glue, see [Special parameters used in AWS Glue jobs](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html) in the _AWS Glue Developer Guide_. + +You can configure the connection to your table bucket through a Spark session in a job or with AWS Glue Studio magics in an interactive session. To use the following examples, replace the `placeholder values` with the information for your own table bucket. + +**Using a PySpark script** + + +Use the following code snippet in a PySpark script to configure a AWS Glue job to connect to your table bucket using the JAR. Replace the placeholder values with the information for your own table bucket. + + + spark = SparkSession.builder.appName("glue-s3-tables") \ + .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \ + .config("spark.sql.defaultCatalog", "s3tablesbucket") \ + .config("spark.sql.catalog.s3tablesbucket", "org.apache.iceberg.spark.SparkCatalog") \ + .config("spark.sql.catalog.s3tablesbucket.catalog-impl", "software.amazon.s3tables.iceberg.S3TablesCatalog") \ + .config("spark.sql.catalog.s3tablesbucket.warehouse", "arn:aws:s3tables:Region:111122223333:bucket/amzn-s3-demo-table-bucket") \ + .getOrCreate() + +**Using an interactive AWS Glue session** + + +If you are using an interactive notebook session with AWS Glue 5.0, specify the same configurations using the `%%configure` magic in a cell prior to code execution. Replace the placeholder values with the information for your own table bucket. +