AWS glue documentation change
Summary
Removed 'Best Practices for Installing additional Python libraries' section and simplified Glue version compatibility table. Removed detailed instructions about packaging Python environments into wheel files.
Security assessment
The changes remove previously documented best practices for dependency management but don't reference any specific security vulnerabilities. While dependency management practices can impact security, there's no explicit mention of security issues or vulnerabilities being addressed in this change.
Diff
diff --git a/glue/latest/dg/aws-glue-programming-python-libraries.md b/glue/latest/dg/aws-glue-programming-python-libraries.md index a00f31f8d..5327a5f38 100644 --- a//glue/latest/dg/aws-glue-programming-python-libraries.md +++ b//glue/latest/dg/aws-glue-programming-python-libraries.md @@ -5 +5 @@ -Installing additional Python modules with pip in AWS Glue 2.0 or later Best Practices for Installing additional Python libraries in AWS Glue Including Python files with PySpark native featuresProgramming scripts that use visual transformsZipping librariesLibraries in notebooksLibraries in DevEndpointsLibraries in jobsAnalyze Python dependenciesPython modules already provided in AWS Glue +Installing additional Python modules with pip in AWS Glue 2.0 or laterIncluding Python files with PySpark native featuresProgramming scripts that use visual transformsZipping librariesLibraries in notebooksLibraries in DevEndpointsLibraries in jobsAnalyze Python dependenciesPython modules already provided in AWS Glue @@ -15,2 +14,0 @@ You can install additional Python modules and libraries for use with AWS Glue ET - * Best Practices for Installing additional Python libraries in AWS Glue - @@ -36,15 +34,8 @@ You can install additional Python modules and libraries for use with AWS Glue ET -Glue Version Compatibility and Installation Methods Glue version | Python version | Base image | glibc version | Supported Installation Methods ----|---|---|---|--- -5.0 | 3.11 | [Amazon Linux 2023 (AL2023)](https://aws.amazon.com/linux/amazon-linux-2023/) | 2.34 | - - * (Recommended) Packaging Python environment into a single wheel file - * Installing additional Python libraries in AWS Glue 5.0 or above using requirements.txt - * Installing additional Python libraries using Wheel - * Installing additional Python modules with pip in AWS Glue 2.0 or later - - -4.0 | 3.10 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 | Best Practices for Installing additional Python libraries in AWS Glue -3.0 | 3.7 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 | Best Practices for Installing additional Python libraries in AWS Glue -2.0 | 3.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue -1.0 | 3.6 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue -0.9 | 2.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 | Best Practices for Installing additional Python libraries in AWS Glue +Glue Version Compatibility and Installation Methods AWS Glue version | Python version | Base image | glibc version +---|---|---|--- +5.0 | 3.11 | [Amazon Linux 2023 (AL2023)](https://aws.amazon.com/linux/amazon-linux-2023/) | 2.34 +4.0 | 3.10 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 +3.0 | 3.7 | [Amazon Linux 2 (AL2)](https://aws.amazon.com/amazon-linux-2/) | 2.26 +2.0 | 3.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 +1.0 | 3.6 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 +0.9 | 2.7 | [Amazon Linux AMI (AL1)](https://aws.amazon.com/amazon-linux-ami/) | 2.17 @@ -134,363 +124,0 @@ When configuring the python modules directly in `--additional-python-modules`, A -## Best Practices for Installing additional Python libraries in AWS Glue - -### (Recommended) Packaging Python environment into a single wheel file - -For a safe and consistent environment, AWS Glue recommends that you snapshot and package your python environment into a wheel file. The benefit of this is that your python environment for reference python modules and its transitive dependencies will be locked. This ensure that your AWS Glue job is not affected when upstream repository like PyPI or dependencies introduces an incompatible updates. - -This file can then be used in your AWS Glue job using the `--additional-python-modules` flag. - -###### Important - -You must run the following script in a similar environment as the AWS Glue version you are running. Please refer to the glue environment details table and ensure you are using the same base OS image and python version. - - - #!/bin/bash - set -e - REQUIREMENTS_FILE="requirements.txt" - FINAL_WHEEL_OUTPUT_DIRECTORY="." - PACKAGE_NAME=$(basename "$(pwd)") - PACKAGE_VERSION="0.1.0" - # Help message - show_help() { - echo "Usage: $0 [options]" - echo "" - echo "Options:" - echo " -r, --requirements FILE Path to requirements.txt file (default: requirements.txt)" - echo " -o, --wheel-output DIR Output directory for final wheel (default: current directory)" - echo " -n, --name NAME Package name (default: current directory name)" - echo " -v, --version VERSION Package version (default: 0.1.0)" - echo " -h, --help Show this help message" - echo " -g, --glue-version Glue version (required)" - echo "" - echo "Example:" - echo " $0 -r custom-requirements.txt -o dist -n my_package -v 1.2.3 -g 4.0" - } - # Parse command line arguments - while [[ $# -gt 0 ]]; do - key="$1" - case $key in - -r | --requirements) - REQUIREMENTS_FILE="$2" - shift 2 - ;; - -o | --wheel-output) - FINAL_WHEEL_OUTPUT_DIRECTORY="$2" - shift 2 - ;; - -n | --name) - PACKAGE_NAME="$2" - shift 2 - ;; - -v | --version) - PACKAGE_VERSION="$2" - shift 2 - ;; - -g | --glue-version) - GLUE_VERSION="$2" - shift 2 - ;; - -h | --help) - show_help - exit 0 - ;; - *) - echo "Unknown option: $1" - show_help - exit 1 - ;; - esac - done - # If package name has dashes, convert to underscores and notify user. We need to check this since we cant import a package with dashes. - if [[ "$PACKAGE_NAME" =~ "-" ]]; then - echo "Warning: Package name '$PACKAGE_NAME' contains dashes. Converting to underscores." - PACKAGE_NAME=$(echo "$PACKAGE_NAME" | tr '-' '_') - fi - UBER_WHEEL_NAME="${PACKAGE_NAME}-${PACKAGE_VERSION}-py3-none-any.whl" - # Check if glue version is provided - if [ -z "$GLUE_VERSION" ]; then - echo "Error: Glue version is required." - exit 1 - fi - # Validate version format (basic check) - if [[ ! "$PACKAGE_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] && [[ ! "$PACKAGE_VERSION" =~ ^[0-9]+\.[0-9]+$ ]]; then - echo "Warning: Version '$PACKAGE_VERSION' doesn't follow semantic versioning (x.y.z or x.y)" - fi - # Check if requirements file exists - if [ ! -f "$REQUIREMENTS_FILE" ]; then - echo "Error: Requirements file '$REQUIREMENTS_FILE' not found." - exit 1 - fi - # Get relevant platform tags/python versions based on glue version - if [[ "$GLUE_VERSION" == "5.0" ]]; then - PYTHON_VERSION="3.11" - GLIBC_VERSION="2.34" - elif [[ "$GLUE_VERSION" == "4.0" ]]; then - PYTHON_VERSION="3.10" - GLIBC_VERSION="2.26" - elif [[ "$GLUE_VERSION" == "3.0" ]]; then - PYTHON_VERSION="3.7" - GLIBC_VERSION="2.26" - elif [[ "$GLUE_VERSION" == "2.0" ]]; then - PYTHON_VERSION="3.7" - GLIBC_VERSION="2.17" - elif [[ "$GLUE_VERSION" == "1.0" ]]; then - PYTHON_VERSION="3.6" - GLIBC_VERSION="2.17" - elif [[ "$GLUE_VERSION" == "0.9" ]]; then - PYTHON_VERSION="2.7" - GLIBC_VERSION="2.17" - else - echo "Error: Unsupported glue version '$GLUE_VERSION'." - exit 1 - fi - echo "Using Glue version $GLUE_VERSION" - echo "Using Glue python version $PYTHON_VERSION" - echo "Using Glue glibc version $GLIBC_VERSION" - PIP_PLATFORM_FLAG="" - is_glibc_compatible() { - # assumes glibc version in the form of major.minor (ex: 2.17) - # glue glibc must be >= platform glibc - local glue_glibc_version="$GLIBC_VERSION" - local platform_glibc_version="$1" - # 2.27 (platform) can run on 2.27 (glue) - if [[ "$platform_glibc_version" == "$glue_glibc_version" ]]; then - return 0 - fi - local glue_glibc_major="${glue_glibc_version%%.*}" - local glue_glibc_minor="${glue_glibc_version#*.}" - local platform_glibc_major="${platform_glibc_version%%.*}" - local platform_glibc_minor="${platform_glibc_version#*.}" - # 3.27 (platform) cannot run on 2.27 (glue) - if [[ "$platform_glibc_major" -gt "$glue_glibc_major" ]]; then - return 1 - fi - # 2.34 (platform) cannot run on 2.27 (glue) - if [[ "$platform_glibc_major" -eq "$glue_glibc_major" ]] && [[ "$platform_glibc_minor" -gt "$glue_glibc_minor" ]]; then - return 1 - fi - # 2.17 (platform) can run on 2.27 (glue) - return 0 - } - PIP_PLATFORM_FLAG="" - if is_glibc_compatible "2.17"; then - PIP_PLATFORM_FLAG="${PIP_PLATFORM_FLAG} --platform manylinux2014_x86_64" - fi - if is_glibc_compatible "2.28"; then - PIP_PLATFORM_FLAG="${PIP_PLATFORM_FLAG} --platform manylinux_2_28_x86_64" - fi - if is_glibc_compatible "2.34"; then - PIP_PLATFORM_FLAG="${PIP_PLATFORM_FLAG} --platform manylinux_2_34_x86_64" - fi - if is_glibc_compatible "2.39"; then - PIP_PLATFORM_FLAG="${PIP_PLATFORM_FLAG} --platform manylinux_2_39_x86_64" - fi - echo "Using pip platform flags: $PIP_PLATFORM_FLAG" - # Convert to absolute paths - REQUIREMENTS_FILE=$(realpath "$REQUIREMENTS_FILE") - FINAL_WHEEL_OUTPUT_DIRECTORY=$(realpath "$FINAL_WHEEL_OUTPUT_DIRECTORY") - TEMP_WORKING_DIR=$(mktemp -d) - VENV_DIR="${TEMP_WORKING_DIR}/.build_venv" - WHEEL_OUTPUT_DIRECTORY="${TEMP_WORKING_DIR}/wheelhouse" - # Cleanup function - cleanup() { - echo "Cleaning up temporary files..." - rm -rf "$TEMP_WORKING_DIR" - }