AWS emr documentation change
Summary
Updated logging documentation with minor grammatical improvements, clarified log storage paths for Spark driver/executor, and adjusted recommendations for CloudWatch logging configurations.
Security assessment
The changes primarily improve clarity and add specific log storage path examples. While permissions for VPC endpoints and S3 buckets are mentioned, these are existing security requirements being reiterated, not new security features or vulnerability fixes. No evidence of addressing a specific security vulnerability.
Diff
diff --git a/emr/latest/EMR-Serverless-UserGuide/logging.md b/emr/latest/EMR-Serverless-UserGuide/logging.md index aa6cdae9c..4796f4091 100644 --- a//emr/latest/EMR-Serverless-UserGuide/logging.md +++ b//emr/latest/EMR-Serverless-UserGuide/logging.md @@ -9 +9 @@ Managed storageAmazon S3Amazon CloudWatch -To monitor your job progress on EMR Serverless and troubleshoot job failures, you can choose how EMR Serverless stores and serves application logs. When you submit a job run, you can specify managed storage, Amazon S3, and Amazon CloudWatch as your logging options. +To monitor your job progress on EMR Serverless and troubleshoot job failures, choose how EMR Serverless stores and serves application logs. When you submit a job run, specify managed storage, Amazon S3, and Amazon CloudWatch as your logging options. @@ -11 +11 @@ To monitor your job progress on EMR Serverless and troubleshoot job failures, yo -With CloudWatch, you can specify the log types and log locations that you want to use, or accept the default types and locations. For more information on CloudWatch logs, see Logging for EMR Serverless with Amazon CloudWatch. With managed storage and S3 logging, the following table shows the log locations and UI availability that you can expect if you choose managed storage, Amazon S3 buckets, or both. +With CloudWatch, specify the log types and log locations that you want to use, or accept the default types and locations. For more information on CloudWatch logs, refer to Logging for EMR Serverless with Amazon CloudWatch. With managed storage and S3 logging, the following table lists the log locations and UI availability that you can expect if you choose managed storage, Amazon S3 buckets, or both. @@ -19 +19 @@ Amazon S3 bucket | Stored in S3 bucket | Stored in S3 bucket | Not supported1 -1 We recommend that you keep the **Managed storage** option selected. Otherwise, you can't use the built-in application UIs. +1 We suggest that you keep the **Managed storage** option selected. Otherwise, you can't use the built-in application UIs. @@ -27 +27 @@ By default, EMR Serverless stores application logs securely in Amazon EMR manage -If you turn off the default option, Amazon EMR can't troubleshoot your jobs on your behalf. Example: You will not be able to view Spark-UI from the EMR Serverless Console. +If you turn off the default option, Amazon EMR can't troubleshoot your jobs on your behalf. Example: You cannot access Spark-UI from the EMR Serverless Console. @@ -42 +42 @@ To turn off this option from the AWS CLI, use the `managedPersistenceMonitoringC -If your EMR Serverless application is in a private subnet with VPC endpoints for Amazon S3 and you attach an endpoint policy to control access, you must add the following permissions for EMR Serverless to store and serve application logs. Replace `Resource` with the `AppInfo` buckets from the available regions table in [Sample policies for private subnets that access Amazon S3](https://docs.aws.amazon.com/emr/latest/ManagementGuide/private-subnet-iampolicy.html#private-subnet-iampolicy-regions). +If your EMR Serverless application is in a private subnet with VPC endpoints for Amazon S3 and you attach an endpoint policy to control access, add the following permissions for EMR Serverless to store and serve application logs. Replace `Resource` with the `AppInfo` buckets from the available regions table in [Sample policies for private subnets that access Amazon S3](https://docs.aws.amazon.com/emr/latest/ManagementGuide/private-subnet-iampolicy.html#private-subnet-iampolicy-regions). @@ -76 +76 @@ JSON -Additionally, you can use the `aws:SourceVpc` condition key to ensure that the request travels through the VPC that the VPC endpoint is attached to. +Additionally, use the `aws:SourceVpc` condition key to ensure that the request travels through the VPC that the VPC endpoint is attached to. @@ -80 +80 @@ Additionally, you can use the `aws:SourceVpc` condition key to ensure that the r -Before your jobs can send log data to Amazon S3, you must include the following permissions in the permissions policy for the job runtime role. Replace ``amzn-s3-demo-logging-bucket`` with the name of your logging bucket. +Before your jobs can send log data to Amazon S3, include the following permissions in the permissions policy for the job runtime role. Replace ``amzn-s3-demo-logging-bucket`` with the name of your logging bucket. @@ -121,0 +122,12 @@ For batch jobs that don't have retries enabled, EMR Serverless sends the logs to +Spark driver logs are stored in the following path by EMR Serverless + + + '/applications/<applicationId>/jobs/<jobId>/SPARK_DRIVER/' + +Spark executor logs are stored in the following path by EMR Serverless + + + '/applications/<applicationId>/jobs/<jobId>/SPARK_EXECUTOR/<EXECUTOR-ID>' + +The <EXECUTOR-ID> is an integer. + @@ -129 +141 @@ EMR Serverless releases 7.1.0 and higher support retry attempts for streaming jo -When you submit a job to an EMR Serverless application, you can choose Amazon CloudWatch as an option to store your application logs. This allows you to use CloudWatch log analysis features such as CloudWatch Logs Insights and Live Tail. You can also stream logs from CloudWatch to other systems such as OpenSearch for further analysis. +When you submit a job to an EMR Serverless application, choose Amazon CloudWatch as an option to store your application logs. This allows you to use CloudWatch log analysis features such as CloudWatch Logs Insights and Live Tail. You can also stream logs from CloudWatch to other systems such as OpenSearch for further analysis. @@ -131 +143 @@ When you submit a job to an EMR Serverless application, you can choose Amazon Cl -EMR Serverless provides real-time logging for driver logs. You can view the logs in real time with the CloudWatch live tail capability, or through CloudWatch CLI tail commands. +EMR Serverless provides real-time logging for driver logs. You can access the logs in real time with the CloudWatch live tail capability, or through CloudWatch CLI tail commands. @@ -133 +145 @@ EMR Serverless provides real-time logging for driver logs. You can view the logs -By default, CloudWatch logging is disabled for EMR Serverless. To enable it, see the configuration in AWS CLI. +By default, CloudWatch logging is disabled for EMR Serverless. To enable it, use the configuration in AWS CLI. @@ -137 +149 @@ By default, CloudWatch logging is disabled for EMR Serverless. To enable it, see -Amazon CloudWatch publishes logs in real time, so it incurs more resources from workers. If you choose a low worker capacity, the impact to your job run time might increase. If you enable CloudWatch logging, we recommend that you choose a greater worker capacity. It's also possible that log publication could throttle if the transactions per second (TPS) rate is too low for `PutLogEvents`. The CloudWatch throttling configuration is global to all services, including EMR Serverless. For more information, see [How do I determine throttling in my CloudWatch logs?](https://repost.aws/knowledge-center/cloudwatch-logs-throttling) on _AWS re:post_. +Amazon CloudWatch publishes logs in real time, so it incurs more resources from workers. If you choose a low worker capacity, the impact to your job run time might increase. If you enable CloudWatch logging, we suggest that you choose a greater worker capacity. It's also possible that log publication could throttle if the transactions per second (TPS) rate is too low for `PutLogEvents`. The CloudWatch throttling configuration is global to all services, including EMR Serverless. For more information, refer to [How do I determine throttling in my CloudWatch logs?](https://repost.aws/knowledge-center/cloudwatch-logs-throttling) on _AWS re:post_. @@ -141 +153 @@ Amazon CloudWatch publishes logs in real time, so it incurs more resources from -Before your jobs can send log data to Amazon CloudWatch, you must include the following permissions in the permissions policy for the job runtime role. +Before your jobs can send log data to Amazon CloudWatch, include the following permissions in the permissions policy for the job runtime role. @@ -182 +194 @@ JSON -To set up Amazon CloudWatch to store logs for EMR Serverless from the AWS CLI, use the `cloudWatchLoggingConfiguration` configuration when you start a job run. To do this, provide the following configuration overrides. Optionally, you can also provide a log group name, log stream prefix name, log types, and an encryption key ARN. +To set up Amazon CloudWatch to store logs for EMR Serverless from the AWS CLI, use the `cloudWatchLoggingConfiguration` configuration when you start a job run. To do this, provide the following configuration overrides. Optionally, also provide a log group name, log stream prefix name, log types, and an encryption key ARN. @@ -191 +203 @@ EMR Serverless releases 7.1.0 and higher support retry attempts for streaming jo -The following shows the minimum configuration that is required to turn on Amazon CloudWatch logging with the default settings for EMR Serverless: +The following demonstrates the minimum configuration that is required to turn on Amazon CloudWatch logging with the default settings for EMR Serverless: @@ -202 +214 @@ The following shows the minimum configuration that is required to turn on Amazon -The following example shows all of the required and optional configurations that you can specify when you turn on Amazon CloudWatch logging for EMR Serverless. The supported `logTypes` values are also listed below this example. +The following example shows all of the required and optional configurations that specify when you turn on Amazon CloudWatch logging for EMR Serverless. The supported `logTypes` values are also listed in the following this example. @@ -219 +231 @@ The following example shows all of the required and optional configurations that -By default, EMR Serverless publishes only the driver stdout and stderr logs to CloudWatch. If you want other logs, then you can specify a container role and corresponding log types with the `logTypes` field. +By default, EMR Serverless publishes only the driver stdout and stderr logs to CloudWatch. If you want other logs, then specify a container role and corresponding log types with the `logTypes` field. @@ -221 +233 @@ By default, EMR Serverless publishes only the driver stdout and stderr logs to C -The following list shows the supported worker types that you can specify for the `logTypes` configuration: +The following list shows the supported worker types that specify for the `logTypes` configuration: