AWS glue documentation change
Summary
Simplified logging architecture documentation, restructured CloudWatch metrics details, and removed IAM permissions section for metrics emission
Security assessment
The changes restructure documentation without addressing vulnerabilities. Removal of IAM permissions guidance could impact security configurations but does not directly fix a disclosed issue. No evidence of addressing a specific security vulnerability in the diff.
Diff
diff --git a/glue/latest/dg/zero-etl-monitoring.md b/glue/latest/dg/zero-etl-monitoring.md index b4a345073..89f257031 100644 --- a//glue/latest/dg/zero-etl-monitoring.md +++ b//glue/latest/dg/zero-etl-monitoring.md @@ -34,22 +34 @@ AWS Glue zero-ETL integrations generate Amazon CloudWatch logs for visibility in -### Log Group Structure - -The logging architecture implements a hierarchical structure of integration instances. Each integration instance maintains dedicated log streams within a centralized log group specifically designed for zero-ETL operations. - -### Log Stream Identification and Path - -Each integration instance is assigned a unique log stream based on its `Integration ID`, extracted from the `Integration ARN`. The complete log stream path follows this structure: - - - Format: /aws-glue/zeroETL-integrations/logs/{IntegrationId} - -**Example** : Given an Integration ARN: - - - arn:aws:glue:us-east-1:123456789012:integration:03cabe77-79e7-4b7a-b3da-8c160bea6bbf - -The corresponding full log stream path would be: - - - /aws-glue/zeroETL-integrations/logs/03cabe77-79e7-4b7a-b3da-8c160bea6bbf - -This architectural approach ensures segregation of log events per integration while maintaining a unified logging framework for the entire zero-ETL infrastructure. The granular logging strategy facilitates efficient troubleshooting, audit compliance, and operational monitoring of data movement processes. +For each integration created, the log events for that integration will be collected under `/aws-glue/zeroETL-integrations/logs/` in Amazon Cloudwatch. Inside the log group, log messages will be split into log streams. Each integration created has a dedicated log stream to where all logs for that integration are written. For example, logs for an integration with IntegrationArn `arn:aws:glue:us-east-1:123456789012:integration:03cabe77-79e7-4b7a-b3da-8c160bea6bbf` can be found under /aws-glue/zeroETL-integrations/logs/03cabe77-79e7-4b7a-b3da-8c160bea6bbf. {IntegrationId} can be referenced from the {integrationArn} generated when an integration is created. @@ -132,11 +111 @@ Log format: zero-ETL integrations emit four types of log messages: -AWS Glue zero-ETL provides real-time operational insights through CloudWatch metrics, enabling proactive monitoring of data integration processes without direct querying of target Iceberg tables. When enabled by adding appropriate permissions on source and target processing role, CloudWatch metrics are automatically emitted to the `AWS/Glue/ZeroETL` namespace after completion of each table ingestion operation. You can setup alarms on your CloudWatch metrics to get notified when a particular Ingestion Job fails. - -### Metric Specifications - -**Ingestion Processing Metrics:** - - * `InsertCount`: Quantifies new records written to target Iceberg table - - * `UpdateCount`: Measures record modifications in target Iceberg table - - * `DeleteCount`: Tracks record removals from target Iceberg table +Once an integration completes, you can see these Amazon Cloudwatch metrics generated in your account for each AWS Glue job run: @@ -143,0 +113 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met +CloudWatch metrics namespace: "AWS/Glue/ZeroETL" @@ -144,0 +115 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met +Metrics dimensions: @@ -145,0 +117 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met + * `integrationArn` @@ -147 +119 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met -**Operational Status Metrics:** + * `loadType` @@ -149 +121 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `IngestionSucceeded`: Binary indicator (1) for successful ingestion completion + * `tableName` @@ -151 +122,0 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `IngestionFailed`: Binary indicator (1) for ingestion failure @@ -153 +123,0 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `LastSyncTimestamp`: Temporal marker indicating the most recent successful source-to-target synchronization @@ -155,0 +126 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met +Metric names: @@ -156,0 +128 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met + * `InsertCount` \- number of records inserted in the target Iceberg table. @@ -158 +130 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met -**Metric Dimensions:** + * `UpdateCount` \- number of records updated in the target Iceberg table. @@ -160 +132 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `integrationArn`: Unique identifier for the integration + * `DeleteCount` \- number of records deleted from the target Iceberg table. @@ -162 +134 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `loadType`: Specifies the event processingType (SEED/CDC) + * `IngestionSucceeded` \- count 1, if the ingestion succeeded for the integration. @@ -164 +136 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met - * `tableName`: Target Iceberg table identifier + * `IngestionFailed` \- count 1, if the ingestion failed for the integration. @@ -165,0 +138 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met + * `LastSyncTimestamp` \- timestamp until which source has been synced to target. @@ -169,23 +141,0 @@ AWS Glue zero-ETL provides real-time operational insights through CloudWatch met -### IAM permissions required to enable CloudWatch Metrics - -When creating your integration, the following IAM permissions are needed by the source and target roles to enable CloudWatch Metrics for an integration. AWS Glue zero-ETL integrations use these permissions provided in the source and target roles to emit CloudWatch Metrics to customer accounts. - - - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "cloudwatch:PutMetricData" - ], - "Resource": "*", - "Condition": { - "StringEquals": { - "cloudwatch:namespace": "AWS/Glue/ZeroETL" - } - } - } - ] - } -