AWS glue documentation change
Summary
Updated AWS Glue getting started example script with improved error handling, resource cleanup, and security best practices including secure temporary file handling and AWS credential validation.
Security assessment
The changes introduce security improvements but don't fix a specific vulnerability. Key security enhancements include: 1) Using openssl for cryptographically secure random ID generation instead of tr/urandom, 2) Setting file permissions (chmod 600) on temporary JSON files, 3) Securely deleting temporary files with shred when available, 4) Adding AWS credential validation via sts get-caller-identity, 5) Adding input validation for JSON files with jq. These are proactive security measures rather than fixes for disclosed vulnerabilities.
Diff
diff --git a/glue/latest/dg/example_glue_GettingStarted_024_section.md b/glue/latest/dg/example_glue_GettingStarted_024_section.md index 337f2bcf3..2f9998634 100644 --- a//glue/latest/dg/example_glue_GettingStarted_024_section.md +++ b//glue/latest/dg/example_glue_GettingStarted_024_section.md @@ -34,0 +35,4 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Cost improvements: Reduced API calls, optimized queries, eliminated redundant operations + # Reliability improvements: Enhanced error handling, input validation, resource tracking + + set -euo pipefail @@ -44 +48 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - UNIQUE_ID=$(LC_ALL=C tr -dc 'a-z0-9' < /dev/urandom | head -c 8) + UNIQUE_ID=$(openssl rand -hex 4) @@ -46,0 +51 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + TABLE_INPUT_FILE="table-input-${UNIQUE_ID}.json" @@ -49 +54,10 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - CREATED_RESOURCES=() + declare -a CREATED_RESOURCES=() + + # Set default region if not provided + AWS_REGION="${AWS_REGION:-us-east-1}" + + # Flag to track if database was successfully created + DATABASE_CREATED=false + + # Trap to ensure cleanup on exit + trap cleanup_resources EXIT @@ -54,2 +68 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "ERROR: $1 failed. Exiting." - cleanup_resources + echo "ERROR: $1 failed." >&2 @@ -61,0 +75 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + local exit_code=$? @@ -74 +88,7 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws glue delete-table --database-name "$DB_NAME" --name "$resource_name" + if [ "$DATABASE_CREATED" = true ]; then + aws glue delete-table \ + --database-name "$DB_NAME" \ + --name "$resource_name" \ + --region "$AWS_REGION" \ + 2>/dev/null || echo "Warning: Failed to delete table $resource_name" + fi @@ -77 +97,4 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws glue delete-database --name "$resource_name" + aws glue delete-database \ + --name "$resource_name" \ + --region "$AWS_REGION" \ + 2>/dev/null || echo "Warning: Failed to delete database $resource_name" @@ -80 +103 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "Unknown resource type: $resource_type" + echo "Unknown resource type: $resource_type" >&2 @@ -84,0 +108,9 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Clean up temporary files securely + if [ -f "$TABLE_INPUT_FILE" ]; then + if command -v shred &> /dev/null; then + shred -vfz -n 3 "$TABLE_INPUT_FILE" 2>/dev/null || rm -f "$TABLE_INPUT_FILE" + else + rm -f "$TABLE_INPUT_FILE" + fi + fi + @@ -85,0 +118,38 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + exit $exit_code + } + + # Function to validate prerequisites + validate_prerequisites() { + # Validate AWS CLI is available + if ! command -v aws &> /dev/null; then + echo "ERROR: AWS CLI is not installed or not in PATH" >&2 + exit 1 + fi + + # Validate AWS CLI version + local AWS_CLI_VERSION + AWS_CLI_VERSION=$(aws --version 2>&1 | cut -d' ' -f1 | cut -d'/' -f2 | cut -d'.' -f1) + if [ "$AWS_CLI_VERSION" -lt 1 ]; then + echo "ERROR: AWS CLI is required" >&2 + exit 1 + fi + + # Validate jq is available for JSON validation + if ! command -v jq &> /dev/null; then + echo "ERROR: jq is not installed or not in PATH" >&2 + exit 1 + fi + + # Validate AWS credentials and get account identity in single call (cost optimization) + local CALLER_IDENTITY + CALLER_IDENTITY=$(aws sts get-caller-identity --region "$AWS_REGION" --query 'Account' --output text 2>/dev/null) || { + echo "ERROR: Failed to get AWS caller identity. Check credentials and permissions." >&2 + exit 1 + } + + if [ -z "$CALLER_IDENTITY" ] || [ "$CALLER_IDENTITY" == "None" ]; then + echo "ERROR: Unable to determine AWS account identity" >&2 + exit 1 + fi + echo "Using AWS Account: $CALLER_IDENTITY" + echo "Using Region: $AWS_REGION" @@ -88 +158,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Step 1: Create a database + # Function to create database with verification + create_database() { @@ -90,2 +161,10 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws glue create-database --database-input "{\"Name\":\"$DB_NAME\",\"Description\":\"Database for AWS Glue tutorial\"}" - check_status "Creating database" + + if ! aws glue create-database \ + --database-input "Name=$DB_NAME,Description=Database for AWS Glue tutorial" \ + --region "$AWS_REGION" \ + --output json > /dev/null 2>&1; then + echo "ERROR: Failed to create database $DB_NAME" >&2 + exit 1 + fi + + DATABASE_CREATED=true @@ -93,0 +173 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + } @@ -95,8 +175,5 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Verify the database was created - echo "Verifying database creation..." - DB_VERIFY=$(aws glue get-database --name "$DB_NAME" --query 'Database.Name' --output text) - check_status "Verifying database" - - if [ "$DB_VERIFY" != "$DB_NAME" ]; then - echo "ERROR: Database verification failed. Expected $DB_NAME but got $DB_VERIFY" - cleanup_resources + # Function to prepare table input JSON + prepare_table_input() { + # Create a temporary JSON file for table input with restricted permissions + if ! touch "$TABLE_INPUT_FILE" 2>/dev/null; then + echo "ERROR: Failed to create temporary file $TABLE_INPUT_FILE" >&2 @@ -105 +181,0 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "Database verification successful." @@ -107,2 +183,5 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Step 2: Create a table - echo "Step 2: Creating a table named $TABLE_NAME in database $DB_NAME" + if ! chmod 600 "$TABLE_INPUT_FILE" 2>/dev/null; then + echo "ERROR: Failed to set permissions on $TABLE_INPUT_FILE" >&2 + rm -f "$TABLE_INPUT_FILE" + exit 1 + fi @@ -110,3 +189 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Create a temporary JSON file for table input - TABLE_INPUT_FILE="table-input-${UNIQUE_ID}.json" - cat > "$TABLE_INPUT_FILE" << EOF + cat > "$TABLE_INPUT_FILE" << 'EOF' @@ -114 +191 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - "Name": "$TABLE_NAME", + "Name": "TABLE_NAME_PLACEHOLDER", @@ -158,4 +235,6 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws glue create-table --database-name "$DB_NAME" --table-input file://"$TABLE_INPUT_FILE" - check_status "Creating table" - CREATED_RESOURCES+=("table:$TABLE_NAME") - echo "Table $TABLE_NAME created successfully." + # Replace placeholder with actual table name + if ! sed -i "s/TABLE_NAME_PLACEHOLDER/$TABLE_NAME/g" "$TABLE_INPUT_FILE" 2>/dev/null; then + echo "ERROR: Failed to substitute table name in JSON file" >&2 + rm -f "$TABLE_INPUT_FILE" + exit 1 + fi @@ -163 +242,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Clean up the temporary file + # Validate JSON syntax before using it + if ! jq empty "$TABLE_INPUT_FILE" 2>/dev/null; then + echo "ERROR: Invalid JSON in table input file" >&2 @@ -164,0 +246,7 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + exit 1 + fi + } + + # Function to create table + create_table() { + echo "Step 2: Creating a table named $TABLE_NAME in database $DB_NAME" @@ -166,4 +254 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Verify the table was created - echo "Verifying table creation..." - TABLE_VERIFY=$(aws glue get-table --database-name "$DB_NAME" --name "$TABLE_NAME" --query 'Table.Name' --output text) - check_status "Verifying table" + prepare_table_input @@ -171,3 +256,7 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - if [ "$TABLE_VERIFY" != "$TABLE_NAME" ]; then - echo "ERROR: Table verification failed. Expected $TABLE_NAME but got $TABLE_VERIFY" - cleanup_resources + if ! aws glue create-table \ + --database-name "$DB_NAME" \ + --table-input "file://${TABLE_INPUT_FILE}" \ + --region "$AWS_REGION" \ + --output json > /dev/null 2>&1; then + echo "ERROR: Failed to create table $TABLE_NAME" >&2 + rm -f "$TABLE_INPUT_FILE" @@ -176 +264,0 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "Table verification successful." @@ -178 +266,6 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Step 3: Get table details + CREATED_RESOURCES+=("table:$TABLE_NAME") + echo "Table $TABLE_NAME created successfully." + } + + # Function to get and display table details