AWS emr documentation change
Summary
Added EMR version 7 example with WebIdentityTokenCredentialsProvider and security-related Spark configurations
Security assessment
Expanded documentation for IRSA (IAM Roles for Service Accounts) authentication patterns. This improves security documentation but does not address a specific disclosed vulnerability.
Diff
diff --git a/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-security-irsa.md b/emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-security-irsa.md index 63da9bd08..01b7d5564 100644 --- a//emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-security-irsa.md +++ b//emr/latest/EMR-on-EKS-DevelopmentGuide/spark-operator-security-irsa.md @@ -107 +107 @@ After you [configure the Kubernetes service account](), you can run a Spark appl - 1. Create a new file `word-count.yaml`, with a `SparkApplication` definition for your word-count application. + 1. Create a new file `word-count.yaml`, with a `SparkApplication` definition for your word-count application, based on Amazon EMR version 6. @@ -137,0 +138,51 @@ After you [configure the Kubernetes service account](), you can run a Spark appl + spark.executor.extraLibraryPath: /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native + sparkVersion: "3.3.1" + restartPolicy: + type: Never + driver: + cores: 1 + coreLimit: "1200m" + memory: "512m" + labels: + version: 3.3.1 + serviceAccount: my-spark-driver-sa + executor: + cores: 1 + instances: 1 + memory: "512m" + labels: + version: 3.3.1 + EOF + + +If you're using the spark operator with a version 7 release, you adjust some of the configuration values: + + cat >word-count.yaml <<EOF + apiVersion: "sparkoperator.k8s.io/v1beta2" + kind: SparkApplication + metadata: + name: word-count + namespace: spark-operator + spec: + type: Java + mode: cluster + image: "895885662937.dkr.ecr.us-west-2.amazonaws.com/spark/emr-7.7.0:latest" + imagePullPolicy: Always + mainClass: org.apache.spark.examples.JavaWordCount + mainApplicationFile: local:///usr/lib/spark/examples/jars/spark-examples.jar + arguments: + - s3://my-pod-bucket/poem.txt + hadoopConf: + # EMRFS filesystem + fs.s3.customAWSCredentialsProvider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider + fs.s3.impl: com.amazon.ws.emr.hadoop.fs.EmrFileSystem + fs.AbstractFileSystem.s3.impl: org.apache.hadoop.fs.s3.EMRFSDelegate + fs.s3.buffer.dir: /mnt/s3 + fs.s3.getObject.initialSocketTimeoutMilliseconds: "2000" + mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem: "2" + mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem: "true" + sparkConf: + # Required for EMR Runtime + spark.driver.extraClassPath: /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/aws-java-sdk-v2/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/* + spark.driver.extraLibraryPath: /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:/docker/usr/lib/hadoop/lib/native:/docker/usr/lib/hadoop-lzo/lib/native + spark.executor.extraClassPath: /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/aws-java-sdk-v2/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/home/hadoop/extrajars/*