AWS emr: Tighten IAM PassRole condition for EMR on EKS Spark Connect
Summary
Updates the Spark Connect setup guide: the IAM policy statement is renamed to PassRoleToEKSPodIdentity and its condition is tightened from StringLike/emr-containers.amazonaws.com to StringEquals/pods.eks.amazonaws.com plus an ArnLike restriction on iam:AssociatedResourceARN scoped to a specific EKS cluster. Also renames CLI parameters (--security-configuration-data, --virtual-cluster-identifier, virtualClusterIdentifier), bumps the PySpark/Spark version for emr-spark-8.1.0 from 4.0.2 to 4.1.1, documents the default of 2 executors, and describes a new one-off Envoy router deployment in the spark-connect-router namespace.
Security assessment
The PassRole condition is narrowed from a wildcard StringLike match on emr-containers.amazonaws.com to an exact StringEquals match on pods.eks.amazonaws.com, and an ArnLike condition on iam:AssociatedResourceARN limits the role to a named EKS cluster. This is least-privilege IAM hardening that reduces the blast radius of an over-broad iam:PassRole grant, but it documents required permissions rather than fixing a specific disclosed vulnerability or incident.
Evidence
+ "iam:PassedToService": "pods.eks.amazonaws.com"
Diff
diff --git a/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-spark-connect.md b/emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-spark-connect.md index 5490de2b2..a33c3f942 100644 --- a//emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-spark-connect.md +++ b//emr/latest/EMR-on-EKS-DevelopmentGuide/emr-eks-spark-connect.md @@ -70 +70 @@ Add the following permissions to your IAM role to create and interact with a Spa - "Sid": "PassRoleToEMRContainers", + "Sid": "PassRoleToEKSPodIdentity", @@ -75,2 +75,7 @@ Add the following permissions to your IAM role to create and interact with a Spa - "StringLike": { - "iam:PassedToService": "emr-containers.amazonaws.com" + "StringEquals": { + "iam:PassedToService": "pods.eks.amazonaws.com" + }, + "ArnLike": { + "iam:AssociatedResourceARN": [ + "arn:aws:eks:region:account-id:cluster/eks-cluster-name" + ] @@ -96 +101 @@ Create a security configuration with a system namespace: - --security-configuration '{ + --security-configuration-data '{ @@ -146 +151 @@ After creating a security configuration, create a Spark Connect managed endpoint -The first Spark Connect endpoint on an EKS cluster takes longer to become `ACTIVE` than subsequent ones. For the first endpoint on the EKS cluster, Amazon EMR on EKS provisions the shared networking components used for connectivity — an internal Network Load Balancer (NLB) and a VPC interface endpoint (AWS PrivateLink) — which can take several minutes. These components are created only once per EKS cluster and are reused by all later endpoints on that cluster, so subsequent endpoints start up faster. +The first Spark Connect endpoint on an EKS cluster takes longer to become `ACTIVE` than subsequent ones. For the first endpoint on the EKS cluster, Amazon EMR on EKS provisions the shared networking components used for connectivity — an internal Network Load Balancer (NLB), a VPC interface endpoint (AWS PrivateLink), and a one-off Envoy router deployment (in the `spark-connect-router` namespace) that routes gRPC traffic to endpoints — which can take several minutes. These components are created only once per EKS cluster and are reused by all later endpoints on that cluster, so subsequent endpoints start up faster. @@ -155 +159,0 @@ The first Spark Connect endpoint on an EKS cluster takes longer to become `ACTIV - --security-configuration-id SECURITY_CONFIGURATION_ID \ @@ -157,0 +162,4 @@ The first Spark Connect endpoint on an EKS cluster takes longer to become `ACTIV +###### Note + +By default, a Spark Connect managed endpoint starts with _2 executors_. If you don't provide configuration overrides, the endpoint uses this default. To run with a different number of executors, set `spark.executor.instances` in the configuration overrides, as shown in the following example. + @@ -167 +174,0 @@ The following example includes Spark configuration overrides with dynamic alloca - --security-configuration-id SECURITY_CONFIGURATION_ID \ @@ -218 +225 @@ The response includes the `authProxyUrl` field when the endpoint is `ACTIVE`. - --virtual-cluster-id VIRTUAL_CLUSTER_ID \ + --virtual-cluster-identifier VIRTUAL_CLUSTER_ID \ @@ -238 +245 @@ Use the `credentials.token` value as the `x-aws-proxy-auth` parameter when conne - 3. Install the PySpark client matching the Spark version on your endpoint (Spark 3.5.8 for `emr-7.14.0`, Spark 4.0.2 for `emr-spark-8.1.0`): + 3. Install the PySpark client matching the Spark version on your endpoint (Spark 3.5.8 for `emr-7.14.0`, Spark 4.1.1 for `emr-spark-8.1.0`): @@ -244 +251 @@ Use the `credentials.token` value as the `x-aws-proxy-auth` parameter when conne - pip install pyspark[connect]==4.0.2 + pip install pyspark[connect]==4.1.1 @@ -291 +298 @@ The following Python script obtains the auth proxy URL and session token, then c - virtualClusterId=VIRTUAL_CLUSTER_ID, + virtualClusterIdentifier=VIRTUAL_CLUSTER_ID, @@ -324 +331 @@ Consider the following when running interactive workloads through Spark Connect - * The PySpark version installed locally must match the Apache Spark version on your endpoint (Spark 3.5.8 for `emr-7.14.0`, Spark 4.0.2 for `emr-spark-8.1.0`). A version mismatch causes connection errors or unexpected behavior. + * The PySpark version installed locally must match the Apache Spark version on your endpoint (Spark 3.5.8 for `emr-7.14.0`, Spark 4.1.1 for `emr-spark-8.1.0`). A version mismatch causes connection errors or unexpected behavior. @@ -334 +341 @@ Consider the following when running interactive workloads through Spark Connect - * Spark Connect endpoints use a Network Load Balancer (NLB) to route gRPC traffic. The NLB is created when the first Spark Connect endpoint is created and is only deleted when the last session-enabled virtual cluster is deleted. You are responsible for NLB costs while it exists, in addition to EKS compute resources consumed by the Spark driver and executors during your session. + * Spark Connect endpoints use a Network Load Balancer (NLB) to route gRPC traffic. The NLB is created when the first Spark Connect endpoint is created and is only deleted when the last session-enabled virtual cluster is deleted. Amazon EMR on EKS also runs a single Envoy router (in the `spark-connect-router` namespace, one per EKS cluster) that routes gRPC traffic to endpoints; it is created with the first Spark Connect endpoint and terminated when the last session-enabled virtual cluster is deleted. You are responsible for NLB costs and the EKS compute consumed by the Envoy router while they exist, in addition to EKS compute resources consumed by the Spark driver and executors during your session.