AWS emr documentation change
Summary
Restructured documentation to clarify required S3 buckets for EMR in private subnets, added version-specific bucket requirements, replaced region table with categorized bucket explanations, and updated example policies to reflect current EMR versions.
Security assessment
The changes enhance security documentation by detailing least-principle bucket permissions for VPC endpoints, but don't address any specific vulnerability. The updates help users configure secure policies by specifying exact bucket requirements per EMR version.
Diff
diff --git a/emr/latest/ManagementGuide/private-subnet-iampolicy.md b/emr/latest/ManagementGuide/private-subnet-iampolicy.md index d2d10d28a..fbb49f329 100644 --- a//emr/latest/ManagementGuide/private-subnet-iampolicy.md +++ b//emr/latest/ManagementGuide/private-subnet-iampolicy.md @@ -7 +7 @@ -Available regions +Required bucketsExample policy @@ -11 +11 @@ Available regions -For private subnets, at a minimum you must provide the ability for Amazon EMR to access Amazon Linux repositories. This private subnet policy is a part of the VPC endpoint policies for accessing Amazon S3. +When launching an Amazon EMR cluster in a private subnet, you must provide a route to Amazon S3. By default, a gateway endpoint for Amazon S3 allows access to all buckets. You can create a VPC endpoint policy to restrict access to specific buckets; if you do, you will need to add policy statements allowing access to the specific S3 buckets required by Amazon EMR. For more information about Amazon S3 endpoints, see [Gateway endpoints for Amazon S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html). @@ -13 +13 @@ For private subnets, at a minimum you must provide the ability for Amazon EMR to -With Amazon EMR 5.25.0 or later, to enable one-click access to persistent Spark history server, you must allow Amazon EMR to access the system bucket that collects Spark event logs. If you enable logging, provide PUT permissions to the following bucket: +It is up to you to determine the policy restrictions that meet your business needs. This page details the buckets required by Amazon EMR to successfully launch a cluster, followed by an example VPC endpoint policy granting access to those buckets. @@ -14,0 +15 @@ With Amazon EMR 5.25.0 or later, to enable one-click access to persistent Spark +## Required buckets @@ -16 +17 @@ With Amazon EMR 5.25.0 or later, to enable one-click access to persistent Spark - aws157-logs-${AWS::Region}/* +### Amazon Linux AMI repositories @@ -18 +19 @@ With Amazon EMR 5.25.0 or later, to enable one-click access to persistent Spark -For more information, see [One-click access to persistent Spark History Server](https://docs.aws.amazon.com/emr/latest/ManagementGuide/app-history-spark-UI.html). +All Amazon EMR clusters require access to Amazon Linux repositories. The specific bucket ARNs depend on the version of Amazon Linux being used, which depends on the Amazon EMR release being used: @@ -20 +21 @@ For more information, see [One-click access to persistent Spark History Server]( -It is up to you to determine the policy restrictions that meet your business needs. The following example policy provides permissions to access Amazon Linux repositories and the Amazon EMR system bucket for collecting Spark event logs. It shows a few sample resource names for the buckets. + * Amazon EMR 5.29.0 and earlier: AL1 repos `arn:aws:s3:::packages.`region`.amazonaws.com` and `arn:aws:s3:::repo.`region`.amazonaws.com` @@ -22 +23 @@ It is up to you to determine the policy restrictions that meet your business nee -For more information about using IAM policies with Amazon VPC endpoints, see [Endpoint policies for Amazon S3](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html#vpc-endpoints-policies-s3). + * Amazon EMR 5.30.0 through 6.15.0: AL2 repos `arn:aws:s3:::amazonlinux.`region`.amazonaws.com` and `arn:aws:s3:::amazonlinux-2-repos-`region`` @@ -24 +25 @@ For more information about using IAM policies with Amazon VPC endpoints, see [En -The following policy example contains sample resources in the us-east-1 region. + * Amazon EMR 7.0.0 and later: AL2023 repo `arn:aws:s3:::al2023-repos-`region`-de612dc2` @@ -26 +26,0 @@ The following policy example contains sample resources in the us-east-1 region. -JSON @@ -29 +28,0 @@ JSON -**** @@ -30,0 +30,19 @@ JSON +### Amazon EMR repositories + +Amazon EMR 5.22.0 and later require access to the EMR repository bucket `arn:aws:s3:::repo.`region`.emr.amazonaws.com`. + +Amazon EMR 8.0.0 and later and Amazon EMR Spark 8.0.0 and later require access to the EMR instance data buckets `arn:aws:s3:::aws157-instance-data-0-prod-`region`` and `arn:aws:s3:::aws157-instance-data-1-prod-`region``. + +In the ap-southeast-2 region, these buckets are instead named `arn:aws:s3:::aws157-instance-data-bucket-0-prod-ap-southeast-2` and `arn:aws:s3:::aws157-instance-data-bucket-1-prod-ap-southeast-2`. + +### Logging + +If you enable cluster logging, you will need PUT permissions for the bucket you specify as the log destination when creating the cluster, as well as the system logs bucket. In the us-east-1 region, the bucket ARN is `arn:aws:s3:::aws157-logs-prod`; for all other regions, the bucket ARN is `arn:aws:s3:::aws157-logs-prod-`region``. + +### Persistent application user interfaces + +With Amazon EMR 5.25.0 or later, to enable one-click access to persistent application user interfaces, you must allow Amazon EMR to access the system bucket that collects application logs, `arn:aws:s3:::prod.`region`.appinfo.src`. For more information, see [View persistent application user interfaces in Amazon EMR](https://docs.aws.amazon.com/emr/latest/ManagementGuide/app-history-spark-UI.html). + +## Example policy + +The following example policy provides the permissions required to launch an Amazon EMR 8.0.0 cluster in a private subnet in the us-east-2 region, with logging and persistent application user interfaces enabled. @@ -37 +55 @@ JSON - "Sid": "AmazonLinuxAMIRepositoryAccess", + "Sid": "AmazonLinux2023AMIRepositoryAccess", @@ -38,0 +57 @@ JSON + "Principal": "*", @@ -43,2 +62 @@ JSON - "arn:aws:s3:::packages.us-east-1.amazonaws.com/*", - "arn:aws:s3:::repo.us-east-1.amazonaws.com/*" + "arn:aws:s3:::al2023-repos-us-east-2-de612dc2/*" @@ -48 +66 @@ JSON - "Sid": "EnableApplicationHistory", + "Sid": "EmrRepositoryAccess", @@ -49,0 +68 @@ JSON + "Principal": "*", @@ -51,5 +70 @@ JSON - "s3:Put*", - "s3:Get*", - "s3:Create*", - "s3:Abort*", - "s3:List*" + "s3:GetObject" @@ -58 +73,3 @@ JSON - "arn:aws:s3:::prod.us-east-1.appinfo.src/*" + "arn:aws:s3:::repo.us-east-2.emr.amazonaws.com/*", + "arn:aws:s3:::aws157-instance-data-0-prod-us-east-2/*", + "arn:aws:s3:::aws157-instance-data-1-prod-us-east-2/*" @@ -60,10 +77 @@ JSON - } - ] - } - - -The following example policy provides the permissions required to access Amazon Linux 2 repositories in the us-east-1 region. - - - { - "Statement": [ + }, @@ -71 +79 @@ The following example policy provides the permissions required to access Amazon - "Sid": "AmazonLinux2AMIRepositoryAccess", + "Sid": "EnableClusterLogs", @@ -74 +82,3 @@ The following example policy provides the permissions required to access Amazon - "Action": "s3:GetObject", + "Action": [ + "s3:Put*" + ], @@ -76,2 +86,2 @@ The following example policy provides the permissions required to access Amazon - "arn:aws:s3:::amazonlinux.us-east-1.amazonaws.com/*", - "arn:aws:s3:::amazonlinux-2-repos-us-east-1/*" + "arn:aws:s3:::aws157-logs-prod-us-east-2/*", + "arn:aws:s3:::my-logs-bucket/*" @@ -79,9 +89 @@ The following example policy provides the permissions required to access Amazon - } - ] - } - -The following example policy provides the permissions required to access Amazon Linux 2023 repositories in the us-east-1 region. - - - { - "Statement": [ + }, @@ -89 +91 @@ The following example policy provides the permissions required to access Amazon - "Sid": "AmazonLinux2023AMIRepositoryAccess", + "Sid": "EnableApplicationHistory", @@ -92 +94,7 @@ The following example policy provides the permissions required to access Amazon - "Action": "s3:GetObject", + "Action": [ + "s3:Put*", + "s3:Get*", + "s3:Create*", + "s3:Abort*", + "s3:List*" + ], @@ -94 +102 @@ The following example policy provides the permissions required to access Amazon - "arn:aws:s3:::al2023-repos-us-east-1-de612dc2/*" + "arn:aws:s3:::prod.us-east-2.appinfo.src/*" @@ -100,39 +107,0 @@ The following example policy provides the permissions required to access Amazon -## Available regions - -The following table contains a list of buckets by region, and includes both an Amazon Resource Name (ARN) for the respository and a string that represents the ARN for the `appinfo.src`. The ARN, or Amazon Resource Name, is a string that uniquely identifies an AWS resource. - -Region | Repository buckets | AppInfo bucket ----|---|--- -_US East (Ohio)_ | "arn:aws:s3:::packages.us-east-2.amazonaws.com/","arn:aws:s3:::repo.us-east-2.amazonaws.com/","arn:aws:s3:::repo.us-east-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.us-east-2.appinfo.src/*" -_US East (N. Virginia)_ | "arn:aws:s3:::packages.us-east-1.amazonaws.com/","arn:aws:s3:::repo.us-east-1.amazonaws.com/","arn:aws:s3:::repo.us-east-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.us-east-1.appinfo.src/*" -_US West (N. California)_ | "arn:aws:s3:::packages.us-west-1.amazonaws.com/","arn:aws:s3:::repo.us-west-1.amazonaws.com/","arn:aws:s3:::repo.us-west-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.us-west-1.appinfo.src/*" -_US West (Oregon)_ | "arn:aws:s3:::packages.us-west-2.amazonaws.com/","arn:aws:s3:::repo.us-west-2.amazonaws.com/","arn:aws:s3:::repo.us-west-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.us-west-2.appinfo.src/*" -_Africa (Cape Town)_ | "arn:aws:s3:::packages.af-south-1.amazonaws.com/","arn:aws:s3:::repo.af-south-1.amazonaws.com/","arn:aws:s3:::repo.af-south-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.af-south-1.appinfo.src/*" -_Africa (Cape Town)_ | "arn:aws:s3:::packages.ap-east-1.amazonaws.com/","arn:aws:s3:::repo.ap-east-1.amazonaws.com/","arn:aws:s3:::repo.ap-east-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-east-1.appinfo.src/*" -_Asia Pacific (Hyderabad)_ | "arn:aws:s3:::packages.ap-south-2.amazonaws.com/","arn:aws:s3:::repo.ap-south-2.amazonaws.com/","arn:aws:s3:::repo.ap-south-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-south-2.appinfo.src/*" -_Asia Pacific (Jakarta)_ | "arn:aws:s3:::packages.ap-southeast-3.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-3.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-3.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-southeast-3.appinfo.src/*" -_Asia Pacific (Malaysia)_ | "arn:aws:s3:::packages.ap-southeast-5.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-5.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-5.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-southeast-5.appinfo.src/*" -_Asia Pacific (Melbourne)_ | "arn:aws:s3:::packages.ap-southeast-4.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-4.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-4.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-south-4.appinfo.src/*" -_Asia Pacific (Mumbai)_ | "arn:aws:s3:::packages.ap-south-1.amazonaws.com/","arn:aws:s3:::repo.ap-south-1.amazonaws.com/","arn:aws:s3:::repo.ap-south-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-south-1.appinfo.src/*" -_Asia Pacific (Osaka)_ | "arn:aws:s3:::packages.ap-northeast-3.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-3.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-3.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-northeast-3.appinfo.src/*" -_Asia Pacific (Seoul)_ | "arn:aws:s3:::packages.ap-northeast-2.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-2.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-northeast-2.appinfo.src/*" -_Asia Pacific (Singapore)_ | "arn:aws:s3:::packages.ap-southeast-1.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-1.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-southeast-1.appinfo.src/*" -_Asia Pacific (Sydney)_ | "arn:aws:s3:::packages.ap-southeast-2.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-2.amazonaws.com/","arn:aws:s3:::repo.ap-southeast-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-southeast-2.appinfo.src/*" -_Asia Pacific (Tokyo)_ | "arn:aws:s3:::packages.ap-northeast-1.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-1.amazonaws.com/","arn:aws:s3:::repo.ap-northeast-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ap-northeast-1.appinfo.src/*" -_Canada (Central)_ | "arn:aws:s3:::packages.ca-central-1.amazonaws.com/","arn:aws:s3:::repo.ca-central-1.amazonaws.com/","arn:aws:s3:::repo.ca-central-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ca-central-1.appinfo.src/*" -_Canada West (Calgary)_ | "arn:aws:s3:::packages.ca-west-1.amazonaws.com/","arn:aws:s3:::repo.ca-west-1.amazonaws.com/","arn:aws:s3:::repo.ca-west-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.ca-west-1.appinfo.src/*" -_Europe (Frankfurt)_ | "arn:aws:s3:::packages.eu-central-1.amazonaws.com/","arn:aws:s3:::repo.eu-central-1.amazonaws.com/","arn:aws:s3:::repo.eu-central-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-central-1.appinfo.src/*" -_Europe (Ireland)_ | "arn:aws:s3:::packages.eu-west-1.amazonaws.com/","arn:aws:s3:::repo.eu-west-1.amazonaws.com/","arn:aws:s3:::repo.eu-west-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-west-1.appinfo.src/*" -_Europe (London)_ | "arn:aws:s3:::packages.eu-west-2.amazonaws.com/","arn:aws:s3:::repo.eu-west-2.amazonaws.com/","arn:aws:s3:::repo.eu-west-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-west-2.appinfo.src/*" -_Europe (Milan)_ | "arn:aws:s3:::packages.eu-south-1.amazonaws.com/","arn:aws:s3:::repo.eu-south-1.amazonaws.com/","arn:aws:s3:::repo.eu-south-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-south-1.appinfo.src/*" -_Europe (Paris)_ | "arn:aws:s3:::packages.eu-west-3.amazonaws.com/","arn:aws:s3:::repo.eu-west-3.amazonaws.com/","arn:aws:s3:::repo.eu-west-3.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-west-3.appinfo.src/*" -_Europe (Spain)_ | "arn:aws:s3:::packages.eu-south-2.amazonaws.com/","arn:aws:s3:::repo.eu-south-2.amazonaws.com/","arn:aws:s3:::repo.eu-south-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-south-2.appinfo.src/*" -_Europe (Stockholm)_ | "arn:aws:s3:::packages.eu-north-1.amazonaws.com/","arn:aws:s3:::repo.eu-north-1.amazonaws.com/","arn:aws:s3:::repo.eu-north-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-north-1.appinfo.src/*" -_Europe (Zurich)_ | "arn:aws:s3:::packages.eu-central-2.amazonaws.com/","arn:aws:s3:::repo.eu-central-2.amazonaws.com/","arn:aws:s3:::repo.eu-central-2.emr.amazonaws.com/*" | "arn:aws:s3:::prod.eu-central-2.appinfo.src/*" -_Israel (Tel Aviv)_ | "arn:aws:s3:::packages.il-central-1.amazonaws.com/","arn:aws:s3:::repo.il-central-1.amazonaws.com/","arn:aws:s3:::repo.il-central-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.il-central-1.appinfo.src/*" -_Middle East (Bahrain)_ | "arn:aws:s3:::packages.me-south-1.amazonaws.com/","arn:aws:s3:::repo.me-south-1.amazonaws.com/","arn:aws:s3:::repo.me-south-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.me-south-1.appinfo.src/*" -_Middle East (UAE)_ | "arn:aws:s3:::packages.me-central-1.amazonaws.com/","arn:aws:s3:::repo.me-central-1.amazonaws.com/","arn:aws:s3:::repo.me-central-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.me-central-1.appinfo.src/*" -_South America (São Paulo)_ | "arn:aws:s3:::packages.sa-east-1.amazonaws.com/","arn:aws:s3:::repo.sa-east-1.amazonaws.com/","arn:aws:s3:::repo.sa-east-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.sa-east-1.appinfo.src/*" -_AWS GovCloud (US-East)_ | "arn:aws:s3:::packages.us-gov-east-1.amazonaws.com/","arn:aws:s3:::repo.us-gov-east-1.amazonaws.com/","arn:aws:s3:::repo.us-gov-east-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.us-gov-east-1.appinfo.src/*" -_AWS GovCloud (US-West)_ | "arn:aws:s3:::packages.us-gov-west-1.amazonaws.com/","arn:aws:s3:::repo.us-gov-west-1.amazonaws.com/","arn:aws:s3:::repo.us-gov-west-1.emr.amazonaws.com/*" | "arn:aws:s3:::prod.me-south-1.appinfo.src/*" -