AWS glue documentation change
Summary
Added comprehensive guidance for Python dependency management including version compatibility tables, security best practices, dependency analysis tools, and a packaging script for consistent environments
Security assessment
The changes emphasize security best practices like version pinning ('--additional-python-modules' with pinned versions), warn about security responsibilities under the shared responsibility model, and provide tools to analyze dependencies - but there's no evidence of addressing a specific existing vulnerability. The updates improve security posture through better dependency management guidance rather than fixing a disclosed issue.
Diff
diff --git a/glue/latest/dg/aws-glue-programming-python-libraries.md b/glue/latest/dg/aws-glue-programming-python-libraries.md index e1f86fca8..a00f31f8d 100644 --- a//glue/latest/dg/aws-glue-programming-python-libraries.md +++ b//glue/latest/dg/aws-glue-programming-python-libraries.md @@ -5 +5 @@ -Installing additional Python libraries in AWS Glue 5.0 using requirements.txtInstalling additional Python modules with pip in AWS Glue 2.0 or laterIncluding Python files with PySpark native featuresProgramming scripts that use visual transformsPython modules already provided in AWS GlueZipping librariesLibraries in notebooksLibraries in DevEndpointsLibraries in jobs +Installing additional Python modules with pip in AWS Glue 2.0 or later Best Practices for Installing additional Python libraries in AWS Glue Including Python files with PySpark native featuresProgramming scripts that use visual transformsZipping librariesLibraries in notebooksLibraries in DevEndpointsLibraries in jobsAnalyze Python dependenciesPython modules already provided in AWS Glue @@ -9 +9 @@ Installing additional Python libraries in AWS Glue 5.0 using requirements.txtIns -AWS Glue lets you install additional Python modules and libraries for use with AWS Glue ETL. +You can install additional Python modules and libraries for use with AWS Glue ETL. For AWS Glue 2.0 and above, AWS Glue uses the Python Package Installer (pip3) to install additional modules used by AWS Glue ETL. AWS Glue provides multiple options to bring the additional Python modules to your AWS Glue job environment. You can use the “—additional-python-modules" parameter to bring in modules using Python wheel files, Requirements file (requirement.txt, AWS Glue 5.0 and above), or a list of comma-separated Python modules. @@ -13,2 +12,0 @@ AWS Glue lets you install additional Python modules and libraries for use with A - * Installing additional Python libraries in AWS Glue 5.0 using requirements.txt - @@ -16,0 +15,2 @@ AWS Glue lets you install additional Python modules and libraries for use with A + * Best Practices for Installing additional Python libraries in AWS Glue + @@ -21,2 +20,0 @@ AWS Glue lets you install additional Python modules and libraries for use with A - * Python modules already provided in AWS Glue - @@ -27 +25 @@ AWS Glue lets you install additional Python modules and libraries for use with A - * Loading Python libraries in a development endpoint + * Loading Python libraries in a development endpoint in AWS Glue 0.9/1.0 @@ -30,0 +29,41 @@ AWS Glue lets you install additional Python modules and libraries for use with A + * Proactively analyze Python dependencies + + * Python modules already provided in AWS Glue + + + + +Glue Version Compatibility and Installation Methods Glue version | Python version | Base image | glibc version | Supported Installation Methods +---|---|---|---|--- +5.0 | 3.11 | [Amazon Linux 2023 (AL2023)](https://aws.amazon.com/linux/amazon-linux-2023/) | 2.34 | + + * (Recommended) Packaging Python environment into a single wheel file + * Installing additional Python libraries in AWS Glue 5.0 or above using requirements.txt + * Installing additional Python libraries using Wheel + * Installing additional Python modules with pip in AWS Glue 2.0 or later + + +4.0 | 3.10 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 | Best Practices for Installing additional Python libraries in AWS Glue +3.0 | 3.7 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 | Best Practices for Installing additional Python libraries in AWS Glue +2.0 | 3.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue +1.0 | 3.6 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue +0.9 | 2.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue + +Under the [AWS shared responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/), you are responsible for the management of additional Python modules, libraries, and their dependencies that you use with your AWS Glue ETL jobs. This includes applying updates and security patches. + +AWS Glue does not support compiling native code in the job environment. However, AWS Glue jobs run within an Amazon-managed Linux environment. You may be able to provide your native dependencies in a compiled form through a Python wheel file. Please refer to above table for AWS Glue version compatibility details. + +If your Python dependencies transitively depend on native, compiled code, you may run against the following limitation: AWS Glue does not support compiling native code in the job environment. However, AWS Glue jobs run within an Amazon-managed Linux environment. You may be able to provide your native dependencies in a compiled form through a wheel distribution. Please refer to above table for AWS Glue version compatibility details. + +###### Important + +Using incompatible dependencies can result in runtime issues, particularly for libraries with native extensions that must match the target environment's architecture and system libraries. Each AWS Glue version runs on a specific Python version with pre-installed libraries and system configurations. + +## Installing additional Python modules with pip in AWS Glue 2.0 or later + +AWS Glue uses the Python Package Installer (pip3) to install additional modules to be used by AWS Glue ETL. You can use the `--additional-python-modules` parameter with a list of comma-separated Python modules to add a new module or change the version of an existing module. You can install custom distributions of a library by uploading the distribution to Amazon S3, then include the path to the Amazon S3 object in your list of modules. + +You can pass additional options to pip3 with the `--python-modules-installer-option` parameter. For example, you could pass `"--upgrade"` to upgrade the packages specified by `"--additional-python-modules"`. For more examples, see [ Building Python modules from a wheel for Spark ETL workloads using AWS Glue 2.0 ](https://aws.amazon.com/blogs/big-data/building-python-modules-from-a-wheel-for-spark-etl-workloads-using-aws-glue-2-0/). + +AWS Glue supports installing custom Python packages using wheel (.whl) files stored in Amazon S3. To include wheel files in your AWS Glue jobs, provide a comma-separated list of your wheel files stored in s3 to the `--additional-python-modules` job parameter. For example: + @@ -31,0 +71 @@ AWS Glue lets you install additional Python modules and libraries for use with A + --additional-python-modules s3://amzn-s3-demo-bucket/path/to/package-1.0.0-py3-none-any.whl,s3://your-bucket/path/to/another-package-2.1.0-cp311-cp311-linux_x86_64.whl @@ -32,0 +73 @@ AWS Glue lets you install additional Python modules and libraries for use with A +This approach also supports when you need custom distributions, or packages with native dependencies that are pre-compiled for the correct operating system. For more examples, see [ Building Python modules from a wheel for Spark ETL workloads using AWS Glue 2.0](https://aws.amazon.com/blogs/big-data/building-python-modules-from-a-wheel-for-spark-etl-workloads-using-aws-glue-2-0/). @@ -34 +75 @@ AWS Glue lets you install additional Python modules and libraries for use with A -## Installing additional Python libraries in AWS Glue 5.0 using requirements.txt +You specify the `--additional-python-modules` in the Job parameters field of the AWS Glue console or by altering the job arguments in the AWS SDK. For more information about setting job parameters, see [Using job parameters in AWS Glue jobs](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html). @@ -49 +90,3 @@ Value: `s3://path_to_requirements.txt` -AWS Glue 5.0 nodes initially load python libraries specified in `requirements.txt`. Here's a sample requirements.txt: +AWS Glue 5.0 nodes initially load python libraries specified in `requirements.txt`. + +Here's a sample requirements.txt: @@ -64 +107,389 @@ AWS Glue 5.0 nodes initially load python libraries specified in `requirements.tx -## Installing additional Python modules with pip in AWS Glue 2.0 or later +###### Important + +Please avoid unpinned library versions in your requirements.txt to ensure you have a reliable and deterministic AWS Glue environment for your jobs. + +When you use wheel for direct dependencies, you can bring in incompatible version of your transitive dependencies if they are not pinned correctly. As a best practice, all library versions should be pinned for consistency in AWS Glue jobs. AWS Glue recommends packaging your python environment into a wheel file to ensure consistency and reliability for your production workload. + +To update or to add a new Python module AWS Glue allows passing `--additional-python-modules` parameter with a list of comma-separated Python modules as values. For example to update/ add scikit-learn module use the following key/value: `"--additional-python-modules", "scikit-learn==0.21.3"`. You have two options to directly configure the python modules. + + * **Pinned Python Module (recommended)** + +`"--additional-python-modules", "scikit-learn==0.21.3,ephem==4.1.6"` + + * **Unpinned Python Module: (Not recommended for Production Workloads)** + +`"--additional-python-modules", "scikit-learn>==0.20.0,ephem>=4.0.0"` + +OR + +`"--additional-python-modules", "scikit-learn,ephem"` + + + + +###### Important + +When configuring the python modules directly in `--additional-python-modules`, AWS Glue recommends to use pinned library versions to ensure consistency in AWS Glue job environment. Using unpinned library versions, pulls the latest version of the python modules, however this can introduce breaking changes or bring in incompatible python module leading to job failure due to python installation failure in AWS Glue job environment. We recommend customers to not use unpinned library versions for production workload. As a best practice, AWS Glue recommends packaging your python environment into a wheel file to ensure consistency and reliability for your production workload. + +## Best Practices for Installing additional Python libraries in AWS Glue + +### (Recommended) Packaging Python environment into a single wheel file + +For a safe and consistent environment, AWS Glue recommends that you snapshot and package your python environment into a wheel file. The benefit of this is that your python environment for reference python modules and its transitive dependencies will be locked. This ensure that your AWS Glue job is not affected when upstream repository like PyPI or dependencies introduces an incompatible updates. + +This file can then be used in your AWS Glue job using the `--additional-python-modules` flag. + +###### Important + +You must run the following script in a similar environment as the AWS Glue version you are running. Please refer to the glue environment details table and ensure you are using the same base OS image and python version. + + + #!/bin/bash + set -e + REQUIREMENTS_FILE="requirements.txt" + FINAL_WHEEL_OUTPUT_DIRECTORY="." + PACKAGE_NAME=$(basename "$(pwd)") + PACKAGE_VERSION="0.1.0" + # Help message + show_help() { + echo "Usage: $0 [options]" + echo "" + echo "Options:" + echo " -r, --requirements FILE Path to requirements.txt file (default: requirements.txt)" + echo " -o, --wheel-output DIR Output directory for final wheel (default: current directory)" + echo " -n, --name NAME Package name (default: current directory name)" + echo " -v, --version VERSION Package version (default: 0.1.0)" + echo " -h, --help Show this help message" + echo " -g, --glue-version Glue version (required)" + echo "" + echo "Example:" + echo " $0 -r custom-requirements.txt -o dist -n my_package -v 1.2.3 -g 4.0" + } + # Parse command line arguments + while [[ $# -gt 0 ]]; do + key="$1" + case $key in + -r | --requirements) + REQUIREMENTS_FILE="$2" + shift 2 + ;; + -o | --wheel-output) + FINAL_WHEEL_OUTPUT_DIRECTORY="$2" + shift 2 + ;; + -n | --name) + PACKAGE_NAME="$2" + shift 2 + ;; + -v | --version) + PACKAGE_VERSION="$2" + shift 2 + ;; + -g | --glue-version) + GLUE_VERSION="$2" + shift 2 + ;; + -h | --help) + show_help + exit 0 + ;; + *) + echo "Unknown option: $1" + show_help + exit 1 + ;; + esac + done + # If package name has dashes, convert to underscores and notify user. We need to check this since we cant import a package with dashes. + if [[ "$PACKAGE_NAME" =~ "-" ]]; then + echo "Warning: Package name '$PACKAGE_NAME' contains dashes. Converting to underscores." + PACKAGE_NAME=$(echo "$PACKAGE_NAME" | tr '-' '_') + fi + UBER_WHEEL_NAME="${PACKAGE_NAME}-${PACKAGE_VERSION}-py3-none-any.whl" + # Check if glue version is provided + if [ -z "$GLUE_VERSION" ]; then + echo "Error: Glue version is required." + exit 1 + fi + # Validate version format (basic check) + if [[ ! "$PACKAGE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] && [[ ! "$PACKAGE_VERSION" =~ ^[0-9]+\.[0-9]+$ ]]; then + echo "Warning: Version '$PACKAGE_VERSION' doesn't follow semantic versioning (x.y.z or x.y)" + fi + # Check if requirements file exists + if [ ! -f "$REQUIREMENTS_FILE" ]; then + echo "Error: Requirements file '$REQUIREMENTS_FILE' not found." + exit 1 + fi + # Get relevant platform tags/python versions based on glue version + if [[ "$GLUE_VERSION" == "5.0" ]]; then + PYTHON_VERSION="3.11" + GLIBC_VERSION="2.34" + elif [[ "$GLUE_VERSION" == "4.0" ]]; then + PYTHON_VERSION="3.10"