AWS ec2 documentation change
Summary
Updated EMR tutorial script with enhanced security practices including strict error handling, secure file permissions, S3 bucket security configurations (encryption, public access blocking), secure data handling, and improved cleanup procedures
Security assessment
The changes add security best practices documentation but don't address a specific security vulnerability. Changes include: 1) Setting strict shell mode with error trapping, 2) Adding file permission controls (chmod 600 for logs, scripts, and data files), 3) Implementing S3 security features (server-side encryption, public access blocking, versioning), 4) Adding AWS credential validation, 5) Secure cleanup of temporary files, 6) Using secure flags for S3 operations (--sse AES256). These are proactive security improvements rather than fixes for existing vulnerabilities.
Diff
diff --git a/ec2/latest/devguide/example_emr_GettingStarted_037_section.md b/ec2/latest/devguide/example_emr_GettingStarted_037_section.md index 1bee6ca6f..6a510e685 100644 --- a//ec2/latest/devguide/example_emr_GettingStarted_037_section.md +++ b//ec2/latest/devguide/example_emr_GettingStarted_037_section.md @@ -35,0 +36 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + set -euo pipefail @@ -37 +38,4 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Set up logging + # Security: Set strict mode and trap errors + trap 'handle_error "Script interrupted or command failed"' ERR + + # Set up logging with secure permissions @@ -38,0 +43,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + touch "$LOG_FILE" + chmod 600 "$LOG_FILE" @@ -48,2 +54,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - if [ -n "$BUCKET_NAME" ]; then echo "- S3 Bucket: $BUCKET_NAME"; fi - if [ -n "$CLUSTER_ID" ]; then echo "- EMR Cluster: $CLUSTER_ID"; fi + if [ -n "${BUCKET_NAME:-}" ]; then echo "- S3 Bucket: $BUCKET_NAME"; fi + if [ -n "${CLUSTER_ID:-}" ]; then echo "- EMR Cluster: $CLUSTER_ID"; fi @@ -60 +66 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "CLEANUP CONFIRMATION" + echo "CLEANUP IN PROGRESS" @@ -62,4 +67,0 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "Do you want to clean up all created resources? (y/n): " - read -r CLEANUP_CHOICE - - if [[ "${CLEANUP_CHOICE,,}" == "y" ]]; then @@ -69 +71 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - if [ -n "$CLUSTER_ID" ]; then + if [ -n "${CLUSTER_ID:-}" ]; then @@ -71 +73 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws emr terminate-clusters --cluster-ids "$CLUSTER_ID" + aws emr terminate-clusters --cluster-ids "$CLUSTER_ID" 2>/dev/null || true @@ -74 +76 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws emr wait cluster-terminated --cluster-id "$CLUSTER_ID" + aws emr wait cluster-terminated --cluster-id "$CLUSTER_ID" 2>/dev/null || true @@ -78,2 +80,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Delete S3 bucket and contents if it exists - if [ -n "$BUCKET_NAME" ]; then + # Delete S3 bucket and contents if it exists and is not shared + if [ -n "${BUCKET_NAME:-}" ] && [ "${BUCKET_IS_SHARED:-false}" != "true" ]; then @@ -81 +83 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws s3 rm "s3://$BUCKET_NAME" --recursive + aws s3 rm "s3://$BUCKET_NAME" --recursive 2>/dev/null || true @@ -84 +86 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws s3 rb "s3://$BUCKET_NAME" + aws s3 rb "s3://$BUCKET_NAME" 2>/dev/null || true @@ -87,4 +89,4 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - echo "Cleanup completed." - else - echo "Cleanup skipped. Resources will remain in your AWS account." - echo "To avoid ongoing charges, remember to manually delete these resources." + # Remove temporary key pair file if created by this script + if [ -f "${KEY_NAME_FILE:-}" ]; then + rm -f "$KEY_NAME_FILE" + echo "Removed temporary key pair file." @@ -91,0 +94,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + + echo "Cleanup completed." @@ -93,0 +98,10 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Validate AWS CLI is installed and configured + if ! command -v aws &> /dev/null; then + handle_error "AWS CLI is not installed" + fi + + # Test AWS credentials + if ! aws sts get-caller-identity > /dev/null 2>&1; then + handle_error "AWS credentials are not configured or invalid" + fi + @@ -98 +113,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - --query 'Stacks[0].Outputs[?OutputKey==`BucketName`].OutputValue' --output text 2>/dev/null) + --query 'Stacks[0].Outputs[?OutputKey==`BucketName`].OutputValue' --output text 2>/dev/null || true) + @@ -105 +121 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - BUCKET_NAME="emr${RANDOM_ID}" + BUCKET_NAME="emr-${RANDOM_ID}" @@ -109 +125 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Create S3 bucket + # Create S3 bucket with security best practices @@ -111,2 +127,21 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws s3 mb "s3://$BUCKET_NAME" || handle_error "Failed to create S3 bucket" - echo "S3 bucket created successfully." + aws s3 mb "s3://$BUCKET_NAME" --region "${AWS_REGION:-us-east-1}" || handle_error "Failed to create S3 bucket" + + # Enable bucket versioning for safety + aws s3api put-bucket-versioning --bucket "$BUCKET_NAME" --versioning-configuration Status=Enabled || true + + # Block public access to bucket + aws s3api put-public-access-block --bucket "$BUCKET_NAME" \ + --public-access-block-configuration \ + "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" || true + + # Enable encryption on bucket + aws s3api put-bucket-encryption --bucket "$BUCKET_NAME" \ + --server-side-encryption-configuration '{ + "Rules": [{ + "ApplyServerSideEncryptionByDefault": { + "SSEAlgorithm": "AES256" + } + }] + }' || true + + echo "S3 bucket created successfully with security best practices." @@ -157,0 +193,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Secure the script file + chmod 600 health_violations.py + @@ -160 +198 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws s3 cp health_violations.py "s3://$BUCKET_NAME/" || handle_error "Failed to upload PySpark script" + aws s3 cp health_violations.py "s3://$BUCKET_NAME/" --sse AES256 || handle_error "Failed to upload PySpark script" @@ -165 +203,7 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - curl -o food_establishment_data.zip https://docs.aws.amazon.com/emr/latest/ManagementGuide/samples/food_establishment_data.zip || handle_error "Failed to download sample data" + curl -sS -o food_establishment_data.zip "https://docs.aws.amazon.com/emr/latest/ManagementGuide/samples/food_establishment_data.zip" || handle_error "Failed to download sample data" + + # Verify downloaded file + if [ ! -f food_establishment_data.zip ] || [ ! -s food_establishment_data.zip ]; then + handle_error "Downloaded file is empty or missing" + fi + @@ -168,0 +213,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Secure the sample data file + chmod 600 food_establishment_data.csv + @@ -171 +218 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws s3 cp food_establishment_data.csv "s3://$BUCKET_NAME/" || handle_error "Failed to upload sample data" + aws s3 cp food_establishment_data.csv "s3://$BUCKET_NAME/" --sse AES256 || handle_error "Failed to upload sample data" @@ -173,0 +221,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + # Clean up sensitive local files + rm -f food_establishment_data.zip health_violations.py + @@ -176 +226 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - aws emr create-default-roles || handle_error "Failed to create default roles" + aws emr create-default-roles 2>/dev/null || true @@ -181 +231 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - KEY_PAIRS=$(aws ec2 describe-key-pairs --query "KeyPairs[*].KeyName" --output text) + KEY_PAIRS=$(aws ec2 describe-key-pairs --query "KeyPairs[*].KeyName" --output text 2>/dev/null || true) @@ -185,3 +235,4 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - KEY_NAME="emr-tutorial-key-$RANDOM_ID" - aws ec2 create-key-pair --key-name "$KEY_NAME" --query "KeyMaterial" --output text > "${KEY_NAME}.pem" - chmod 400 "${KEY_NAME}.pem" + KEY_NAME="emr-tutorial-key-${RANDOM_ID}" + KEY_NAME_FILE="${KEY_NAME}.pem" + aws ec2 create-key-pair --key-name "$KEY_NAME" --query "KeyMaterial" --output text > "$KEY_NAME_FILE" + chmod 400 "$KEY_NAME_FILE" @@ -195 +246 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Launch EMR cluster + # Launch EMR cluster with security best practices @@ -205 +256,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - --log-uri "s3://$BUCKET_NAME/logs/") + --log-uri "s3://$BUCKET_NAME/logs/" \ + --ebs-root-volume-size 100 \ + --security-configuration "EMR-Tutorial-SecurityConfig" 2>/dev/null || true) @@ -212,4 +265,9 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Extract cluster ID - CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | grep -o '"ClusterId": "[^"]*' | cut -d'"' -f4) - if [ -z "$CLUSTER_ID" ]; then - handle_error "Failed to extract cluster ID from response" + # Extract cluster ID using jq if available, otherwise use alternative parsing + if command -v jq &> /dev/null; then + CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | jq -r '.ClusterId // empty') + else + CLUSTER_ID=$(echo "$CLUSTER_RESPONSE" | grep -o '"ClusterId"[[:space:]]*:[[:space:]]*"[^"]*' | grep -o 'j-[A-Z0-9]*' || true) + fi + + if [ -z "$CLUSTER_ID" ] || [ "$CLUSTER_ID" == "null" ]; then + handle_error "Failed to extract cluster ID from response: $CLUSTER_RESPONSE" @@ -227,0 +286,2 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + WAIT_COUNT=0 + MAX_WAIT=120 @@ -228,0 +289,3 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + if [ $WAIT_COUNT -ge $MAX_WAIT ]; then + handle_error "Cluster did not reach WAITING state within timeout period" + fi @@ -236,0 +300 @@ There's more on GitHub. Find the complete example and learn how to set up and ru + WAIT_COUNT=$((WAIT_COUNT + 1)) @@ -253,2 +317 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # FIXED: Check if jq is available before using it - # Extract step ID using the appropriate method based on available tools + # Extract step ID using appropriate method @@ -256,3 +319 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Use jq if available - echo "Using jq to parse JSON response" - STEP_ID=$(echo "$STEP_RESPONSE" | jq -r '.StepIds[0]') + STEP_ID=$(echo "$STEP_RESPONSE" | jq -r '.StepIds[0] // empty') @@ -260,15 +321 @@ There's more on GitHub. Find the complete example and learn how to set up and ru - # Fallback to grep/awk if jq is not available - echo "jq not found, using grep for parsing" - STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*"' | grep -o 's-[A-Z0-9]*') - if [ -z "$STEP_ID" ]; then - # Another fallback method - STEP_ID=$(echo "$STEP_RESPONSE" | grep -o '"StepIds":\s*\[\s*"[^"]*' | grep -o 's-[A-Z0-9]*') - if [ -z "$STEP_ID" ]; then - # One more attempt with a different pattern - STEP_ID=$(echo "$STEP_RESPONSE" | grep -o 's-[A-Z0-9]*') - if [ -z "$STEP_ID" ]; then - echo "Full step response: $STEP_RESPONSE" - handle_error "Failed to extract step ID from response" - fi