AWS glue documentation change
Summary
Updated documentation for AWS Glue 5.0 local development with Docker, including ECR image locations, component changes, setup instructions, and removal of deprecated sections
Security assessment
Changes focus on updating development environment setup and Docker image references without addressing security vulnerabilities or introducing new security features. Credential configuration guidance remains standard practice
Diff
diff --git a/glue/latest/dg/aws-glue-programming-etl-libraries.md b/glue/latest/dg/aws-glue-programming-etl-libraries.md index de73fdb59..3688cbea5 100644 --- a/glue/latest/dg/aws-glue-programming-etl-libraries.md +++ b/glue/latest/dg/aws-glue-programming-etl-libraries.md @@ -5 +5 @@ -Developing using AWS Glue StudioDeveloping using interactive sessionsDeveloping using a Docker imageDeveloping using the AWS Glue ETL library +Developing using AWS Glue StudioDeveloping using interactive sessionsDeveloping AWS Glue jobs locally with Docker @@ -29,4 +28,0 @@ When you develop and test your AWS Glue for Spark job scripts, there are multipl - * AWS Glue Studio ETL library - - * Local development - @@ -54 +50 @@ Interactive sessions allow you to build and test applications from the environme -## Developing using a Docker image +## Develop and test AWS Glue jobs locally using a Docker image @@ -56 +52 @@ Interactive sessions allow you to build and test applications from the environme -###### Note +For a production-ready data platform, the development process and CI/CD pipeline for AWS Glue jobs is a key topic. You can flexibly develop and test AWS Glue jobs in a Docker container. AWS Glue hosts Docker images on Docker Hub to set up your development environment with additional utilities. You can use your preferred IDE, notebook, or REPL using AWS Glue ETL library. This topic describes how to develop and test AWS Glue version 5.0 jobs in a Docker container using a Docker image. @@ -58 +54 @@ Interactive sessions allow you to build and test applications from the environme -The instructions in this section have not been tested on Microsoft Windows operating systems. +### Available Docker images @@ -60 +56 @@ The instructions in this section have not been tested on Microsoft Windows opera -For local development and testing on Windows platforms, see the blog [Building an AWS Glue ETL pipeline locally without an AWS account](https://aws.amazon.com/blogs/big-data/building-an-aws-glue-etl-pipeline-locally-without-an-aws-account/) +The following Docker images are available for AWS Glue on [Amazon ECR:](https://gallery.ecr.aws/glue/aws-glue-libs). @@ -62 +58 @@ For local development and testing on Windows platforms, see the blog [Building a -For a production-ready data platform, the development process and CI/CD pipeline for AWS Glue jobs is a key topic. You can flexibly develop and test AWS Glue jobs in a Docker container. AWS Glue hosts Docker images on Docker Hub to set up your development environment with additional utilities. You can use your preferred IDE, notebook, or REPL using AWS Glue ETL library. This topic describes how to develop and test AWS Glue version 4.0 jobs in a Docker container using a Docker image. + * For AWS Glue version 5.0: `public.ecr.aws/glue/aws-glue-libs:5` @@ -64 +60 @@ For a production-ready data platform, the development process and CI/CD pipeline -The following Docker images are available for AWS Glue on Docker Hub. + * For AWS Glue version 4.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_4.0.0_image_01` @@ -66 +62 @@ The following Docker images are available for AWS Glue on Docker Hub. - * For AWS Glue version 4.0: `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` + * For AWS Glue version 3.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_3.0.0_image_01` @@ -68 +64 @@ The following Docker images are available for AWS Glue on Docker Hub. - * For AWS Glue version 3.0: `amazon/aws-glue-libs:glue_libs_3.0.0_image_01` + * For AWS Glue version 2.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_2.0.0_image_01` @@ -70 +65,0 @@ The following Docker images are available for AWS Glue on Docker Hub. - * For AWS Glue version 2.0: `amazon/aws-glue-libs:glue_libs_2.0.0_image_01` @@ -73,0 +69 @@ The following Docker images are available for AWS Glue on Docker Hub. +###### Note @@ -75,3 +71 @@ The following Docker images are available for AWS Glue on Docker Hub. -These images are for x86_64. It is recommended that you test on this architecture. However, it may be possible to rework a local development solution on unsupported base images. - -This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and running the container on a local machine. This container image has been tested for an AWS Glue version 3.3 Spark jobs. This image contains the following: +AWS Glue Docker images are compatible with both x86_64 and arm64. @@ -79 +73 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Amazon Linux +In this example, we use `public.ecr.aws/glue/aws-glue-libs:5` and run the container on a local machine (Mac, Windows, or Linux). This container image has been tested for AWS Glue version 5.0 Spark jobs. The image contains the following: @@ -81 +75 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * AWS Glue ETL library ([aws-glue-libs](https://github.com/awslabs/aws-glue-libs)) + * Amazon Linux 2023 @@ -83 +77 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Apache Spark 3.3.0 + * AWS Glue ETL Library @@ -85 +79 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Spark history server + * Apache Spark 3.5.2 @@ -87 +81 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Jupyter Lab + * Open table format libraries; Apache Iceberg 1.6.1, Apache Hudi 0.15.0, and Delta Lake 3.2.1 @@ -89 +83 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Livy + * AWS Glue Data Catalog Client @@ -91 +85 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and - * Other library dependencies (the same set as the ones of AWS Glue job system) + * Amazon Redshift connector for Apache Spark @@ -92,0 +87 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and + * Amazon DynamoDB connector for Apache Hadoop @@ -96 +90,0 @@ This example describes using `amazon/aws-glue-libs:glue_libs_4.0.0_image_01` and -Complete one of the following sections according to your requirements: @@ -98 +92 @@ Complete one of the following sections according to your requirements: - * Set up the container to use spark-submit +To set up your container, pull the image from ECR Public Gallery and then run the container. This topic demonstrates how to run your container with the following methods, depending on your requirements: @@ -100 +94 @@ Complete one of the following sections according to your requirements: - * Set up the container to use REPL shell (PySpark) + * `spark-submit` @@ -102 +96 @@ Complete one of the following sections according to your requirements: - * Set up the container to use Pytest + * REPL shell `(pyspark)` @@ -104 +98 @@ Complete one of the following sections according to your requirements: - * Set up the container to use Jupyter Lab + * `pytest` @@ -106 +100 @@ Complete one of the following sections according to your requirements: - * Set up the container to use Visual Studio Code + * Visual Studio Code @@ -117 +111 @@ For more information about restrictions when developing AWS Glue code locally, s -### Configuring AWS +#### Configuring AWS @@ -121 +115 @@ To enable AWS API calls from the container, set up AWS credentials by following - 1. Set up the AWS CLI, configuring a named profile. For more information about AWS CLI configuration, see [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) in the AWS CLI documentation. + 1. [ Create an AWS named profile ](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). @@ -123 +117 @@ To enable AWS API calls from the container, set up AWS credentials by following - 2. Run the following command in a terminal: + 2. Open `cmd` on Windows or a terminal on Mac/Linux and run the following command in a terminal: @@ -130,3 +124 @@ To enable AWS API calls from the container, set up AWS credentials by following -You may also need to set the AWS_REGION environment variable to specify the AWS Region to send requests to. - -### Setting up and running the container +In the following sections, we use this AWS named profile. @@ -134 +126 @@ You may also need to set the AWS_REGION environment variable to specify the AWS -Setting up the container to run PySpark code through the spark-submit command includes the following high-level steps: +#### @@ -136 +128 @@ Setting up the container to run PySpark code through the spark-submit command in - 1. Pull the image from Docker Hub. +If you’re running Docker on Windows, choose the Docker icon (right-click) and choose **Switch to Linux containers** before pulling the image. @@ -138 +130 @@ Setting up the container to run PySpark code through the spark-submit command in - 2. Run the container. +Run the following command to pull the image from ECR Public: @@ -140,0 +133 @@ Setting up the container to run PySpark code through the spark-submit command in + docker pull public.ecr.aws/glue/aws-glue-libs:5 @@ -142,9 +135 @@ Setting up the container to run PySpark code through the spark-submit command in - -#### Pulling the image from Docker Hub - -Run the following command to pull the image from Docker Hub: - - - docker pull amazon/aws-glue-libs:glue_libs_4.0.0_image_01 - -#### Running the container +### Run the container @@ -154 +139 @@ You can now run a container using this image. You can choose any of following ba -##### spark-submit +#### spark-submit @@ -158 +143 @@ You can run an AWS Glue job script by running the `spark-submit` command on the - 1. Write the script and save it as `sample1.py` under the `/local_path_to_workspace` directory. Sample code is included as the appendix in this topic. + 1. Write your script and save it as `sample.py` in the example below and save it under the `/local_path_to_workspace/src/` directory using the following commands: @@ -164,0 +150,3 @@ You can run an AWS Glue job script by running the `spark-submit` command on the + 2. These variables are used in the docker run command below. The sample code (sample.py) used in the spark-submit command below is included in the appendix at the end of this topic. + +Run the following command to execute the `spark-submit` command on the container to submit a new Spark application: @@ -166,36 +154,8 @@ You can run an AWS Glue job script by running the `spark-submit` command on the - 2. Run the following command to execute the `spark-submit` command on the container to submit a new Spark application: - - $ docker run -it -v ~/.aws:/home/glue_user/.aws -v $WORKSPACE_LOCATION:/home/glue_user/workspace/ -e AWS_PROFILE=$PROFILE_NAME -e DISABLE_SSL=true --rm -p 4040:4040 -p 18080:18080 --name glue_spark_submit amazon/aws-glue-libs:glue_libs_4.0.0_image_01 spark-submit /home/glue_user/workspace/src/$SCRIPT_FILE_NAME - ...22/01/26 09:08:55 INFO DAGScheduler: Job 0 finished: fromRDD at DynamicFrame.scala:305, took 3.639886 s - root - |-- family_name: string - |-- name: string - |-- links: array - | |-- element: struct - | | |-- note: string - | | |-- url: string - |-- gender: string - |-- image: string - |-- identifiers: array - | |-- element: struct - | | |-- scheme: string - | | |-- identifier: string - |-- other_names: array - | |-- element: struct - | | |-- lang: string - | | |-- note: string - | | |-- name: string - |-- sort_name: string - |-- images: array - | |-- element: struct - | | |-- url: string - |-- given_name: string - |-- birth_date: string - |-- id: string - |-- contact_details: array - | |-- element: struct - | | |-- type: string - | | |-- value: string - |-- death_date: string - - ... + $ docker run -it --rm \ + -v ~/.aws:/home + /hadoop/.aws \ + -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \ + -e AWS_PROFILE=$PROFILE_NAME \ + --name glue5_spark_submit \ + public.ecr.aws/glue/aws-glue-libs:5 \ + spark-submit /home/hadoop/workspace/src/$SCRIPT_FILE_NAME @@ -208 +168,3 @@ You can run an AWS Glue job script by running the `spark-submit` command on the -##### REPL shell (Pyspark) +#### REPL shell (Pyspark) + +You can run REPL (`read-eval-print loops`) shell for interactive development. Run the following command to execute the PySpark command on the container to start the REPL shell: @@ -210 +171,0 @@ You can run an AWS Glue job script by running the `spark-submit` command on the -You can run REPL (read-eval-print loops) shell for interactive development. @@ -212 +173,6 @@ You can run REPL (read-eval-print loops) shell for interactive development. -Run the following command to execute the PySpark command on the container to start the REPL shell: + $ docker run -it --rm \ + -v ~/.aws:/home/hadoop/.aws \ + -e AWS_PROFILE=$PROFILE_NAME \ + --name glue5_pyspark \ + public.ecr.aws/glue/aws-glue-libs:5 \ + pyspark @@ -213,0 +180 @@ Run the following command to execute the PySpark command on the container to sta +You will see the following output: @@ -215,2 +182,6 @@ Run the following command to execute the PySpark command on the container to sta - $ docker run -it -v ~/.aws:/home/glue_user/.aws -e AWS_PROFILE=$PROFILE_NAME -e DISABLE_SSL=true --rm -p 4040:4040 -p 18080:18080 --name glue_pyspark amazon/aws-glue-libs:glue_libs_4.0.0_image_01 pyspark - ... + + Python 3.11.6 (main, Jan 9 2025, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux + Type "help", "copyright", "credits" or "license" for more information. + Setting default log level to "WARN".