AWS glue documentation change
Summary
Removed detailed documentation about developing AWS Glue jobs locally using Docker containers, including Docker image versions, configuration steps, and integration with Visual Studio Code. Also removed appendix about adding JDBC drivers.
Security assessment
The changes remove technical documentation about local Docker development but show no evidence of addressing security vulnerabilities. The removal appears related to feature deprecation or documentation restructuring rather than security fixes. No security advisories or vulnerability mentions are present in the diff.
Diff
diff --git a/glue/latest/dg/aws-glue-programming-etl-libraries.md b/glue/latest/dg/aws-glue-programming-etl-libraries.md index 02f02d4a3..7c9155210 100644 --- a//glue/latest/dg/aws-glue-programming-etl-libraries.md +++ b//glue/latest/dg/aws-glue-programming-etl-libraries.md @@ -5 +5 @@ -Developing using AWS Glue StudioDeveloping using interactive sessionsDeveloping AWS Glue jobs locally with Docker +Developing using AWS Glue StudioDeveloping using interactive sessions @@ -50,316 +49,0 @@ Interactive sessions allow you to build and test applications from the environme -## Develop and test AWS Glue jobs locally using a Docker image - -For a production-ready data platform, the development process and CI/CD pipeline for AWS Glue jobs is a key topic. You can flexibly develop and test AWS Glue jobs in a Docker container. AWS Glue hosts Docker images on Docker Hub to set up your development environment with additional utilities. You can use your preferred IDE, notebook, or REPL using AWS Glue ETL library. This topic describes how to develop and test AWS Glue version 5.0 jobs in a Docker container using a Docker image. - -### Available Docker images - -The following Docker images are available for AWS Glue on [Amazon ECR:](https://gallery.ecr.aws/glue/aws-glue-libs). - - * For AWS Glue version 5.0: `public.ecr.aws/glue/aws-glue-libs:5` - - * For AWS Glue version 4.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_4.0.0_image_01` - - * For AWS Glue version 3.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_3.0.0_image_01` - - * For AWS Glue version 2.0: `public.ecr.aws/glue/aws-glue-libs:glue_libs_2.0.0_image_01` - - - - -###### Note - -AWS Glue Docker images are compatible with both x86_64 and arm64. - -In this example, we use `public.ecr.aws/glue/aws-glue-libs:5` and run the container on a local machine (Mac, Windows, or Linux). This container image has been tested for AWS Glue version 5.0 Spark jobs. The image contains the following: - - * Amazon Linux 2023 - - * AWS Glue ETL Library - - * Apache Spark 3.5.4 - - * Open table format libraries; Apache Iceberg 1.7.1, Apache Hudi 0.15.0, and Delta Lake 3.3.0 - - * AWS Glue Data Catalog Client - - * Amazon Redshift connector for Apache Spark - - * Amazon DynamoDB connector for Apache Hadoop - - - - -To set up your container, pull the image from ECR Public Gallery and then run the container. This topic demonstrates how to run your container with the following methods, depending on your requirements: - - * `spark-submit` - - * REPL shell `(pyspark)` - - * `pytest` - - * Visual Studio Code - - - - -### Prerequisites - -Before you start, make sure that Docker is installed and the Docker daemon is running. For installation instructions, see the Docker documentation for [Mac](https://docs.docker.com/docker-for-mac/install/) or [Linux](https://docs.docker.com/engine/install/). The machine running the Docker hosts the AWS Glue container. Also make sure that you have at least 7 GB of disk space for the image on the host running the Docker. - -For more information about restrictions when developing AWS Glue code locally, see [ Local development restrictions ](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-libraries.html#local-dev-restrictions). - -#### Configuring AWS - -To enable AWS API calls from the container, set up AWS credentials by following steps. In the following sections, we will use this AWS named profile. - - 1. [ Create an AWS named profile ](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). - - 2. Open `cmd` on Windows or a terminal on Mac/Linux and run the following command in a terminal: - - PROFILE_NAME="<your_profile_name>" - - - - -In the following sections, we use this AWS named profile. - -#### - -If you’re running Docker on Windows, choose the Docker icon (right-click) and choose **Switch to Linux containers** before pulling the image. - -Run the following command to pull the image from ECR Public: - - - docker pull public.ecr.aws/glue/aws-glue-libs:5 - -### Run the container - -You can now run a container using this image. You can choose any of following based on your requirements. - -#### spark-submit - -You can run an AWS Glue job script by running the `spark-submit` command on the container. - - 1. Write your script and save it as `sample.py` in the example below and save it under the `/local_path_to_workspace/src/` directory using the following commands: - - $ WORKSPACE_LOCATION=/local_path_to_workspace - $ SCRIPT_FILE_NAME=sample.py - $ mkdir -p ${WORKSPACE_LOCATION}/src - $ vim ${WORKSPACE_LOCATION}/src/${SCRIPT_FILE_NAME} - - 2. These variables are used in the docker run command below. The sample code (sample.py) used in the spark-submit command below is included in the appendix at the end of this topic. - -Run the following command to execute the `spark-submit` command on the container to submit a new Spark application: - - $ docker run -it --rm \ - -v ~/.aws:/home - /hadoop/.aws \ - -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \ - -e AWS_PROFILE=$PROFILE_NAME \ - --name glue5_spark_submit \ - public.ecr.aws/glue/aws-glue-libs:5 \ - spark-submit /home/hadoop/workspace/src/$SCRIPT_FILE_NAME - - 3. (Optionally) Configure `spark-submit` to match your environment. For example, you can pass your dependencies with the `--jars` configuration. For more information, consult [Dynamically Loading Spark Properties](https://spark.apache.org/docs/latest/configuration.html) in the Spark documentation. - - - - -#### REPL shell (Pyspark) - -You can run REPL (`read-eval-print loops`) shell for interactive development. Run the following command to execute the PySpark command on the container to start the REPL shell: - - - $ docker run -it --rm \ - -v ~/.aws:/home/hadoop/.aws \ - -e AWS_PROFILE=$PROFILE_NAME \ - --name glue5_pyspark \ - public.ecr.aws/glue/aws-glue-libs:5 \ - pyspark - -You will see the following output: - - - Python 3.11.6 (main, Jan 9 2025, 00:00:00) [GCC 11.4.1 20230605 (Red Hat 11.4.1-2)] on linux - Type "help", "copyright", "credits" or "license" for more information. - Setting default log level to "WARN". - To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). - Welcome to - ____ __ - / __/__ ___ _____/ /__ - _\ \/ _ \/ _ `/ __/ '_/ - /__ / .__/\_,_/_/ /_/\_\ version 3.5.4-amzn-0 - /_/ - - Using Python version 3.11.6 (main, Jan 9 2025 00:00:00) - Spark context Web UI available at None - Spark context available as 'sc' (master = local[*], app id = local-1740643079929). - SparkSession available as 'spark'. - >>> - -With this REPL shell, you can code and test interactively. - -#### Pytest - -For unit testing, you can use `pytest` for AWS Glue Spark job scripts. Run the following commands for preparation. - - - $ WORKSPACE_LOCATION=/local_path_to_workspace - $ SCRIPT_FILE_NAME=sample.py - $ UNIT_TEST_FILE_NAME=test_sample.py - $ mkdir -p ${WORKSPACE_LOCATION}/tests - $ vim ${WORKSPACE_LOCATION}/tests/${UNIT_TEST_FILE_NAME} - -Run the following command to run `pytest` using `docker run`: - - - $ docker run -i --rm \ - -v ~/.aws:/home/hadoop/.aws \ - -v $WORKSPACE_LOCATION:/home/hadoop/workspace/ \ - --workdir /home/hadoop/workspace \ - -e AWS_PROFILE=$PROFILE_NAME \ - --name glue5_pytest \ - public.ecr.aws/glue/aws-glue-libs:5 \ - -c "python3 -m pytest --disable-warnings" - -Once `pytest` finishes executing unit tests, your output will look something like this: - - - ============================= test session starts ============================== - platform linux -- Python 3.11.6, pytest-8.3.4, pluggy-1.5.0 - rootdir: /home/hadoop/workspace - plugins: integration-mark-0.2.0 - collected 1 item - - tests/test_sample.py . [100%] - - ======================== 1 passed, 1 warning in 34.28s ========================= - -#### Setting up the container to use Visual Studio Code - -To set up the container with Visual Studio Code, complete the following steps: -