AWS glue documentation change
Summary
Expanded logging documentation, added CloudWatch metrics specifications, and IAM permissions required for metrics emission
Security assessment
Documents security-related IAM permissions (cloudwatch:PutMetricData with namespace condition) but does not address a specific vulnerability
Diff
diff --git a/glue/latest/dg/zero-etl-monitoring.md b/glue/latest/dg/zero-etl-monitoring.md index 89f257031..b4a345073 100644 --- a//glue/latest/dg/zero-etl-monitoring.md +++ b//glue/latest/dg/zero-etl-monitoring.md @@ -34 +34,22 @@ AWS Glue zero-ETL integrations generate Amazon CloudWatch logs for visibility in -For each integration created, the log events for that integration will be collected under `/aws-glue/zeroETL-integrations/logs/` in Amazon Cloudwatch. Inside the log group, log messages will be split into log streams. Each integration created has a dedicated log stream to where all logs for that integration are written. For example, logs for an integration with IntegrationArn `arn:aws:glue:us-east-1:123456789012:integration:03cabe77-79e7-4b7a-b3da-8c160bea6bbf` can be found under /aws-glue/zeroETL-integrations/logs/03cabe77-79e7-4b7a-b3da-8c160bea6bbf. {IntegrationId} can be referenced from the {integrationArn} generated when an integration is created. +### Log Group Structure + +The logging architecture implements a hierarchical structure of integration instances. Each integration instance maintains dedicated log streams within a centralized log group specifically designed for zero-ETL operations. + +### Log Stream Identification and Path + +Each integration instance is assigned a unique log stream based on its `Integration ID`, extracted from the `Integration ARN`. The complete log stream path follows this structure: + + + Format: /aws-glue/zeroETL-integrations/logs/{IntegrationId} + +**Example** : Given an Integration ARN: + + + arn:aws:glue:us-east-1:123456789012:integration:03cabe77-79e7-4b7a-b3da-8c160bea6bbf + +The corresponding full log stream path would be: + + + /aws-glue/zeroETL-integrations/logs/03cabe77-79e7-4b7a-b3da-8c160bea6bbf + +This architectural approach ensures segregation of log events per integration while maintaining a unified logging framework for the entire zero-ETL infrastructure. The granular logging strategy facilitates efficient troubleshooting, audit compliance, and operational monitoring of data movement processes. @@ -111 +132,11 @@ Log format: zero-ETL integrations emit four types of log messages: -Once an integration completes, you can see these Amazon Cloudwatch metrics generated in your account for each AWS Glue job run: +AWS Glue zero-ETL provides real-time operational insights through CloudWatch metrics, enabling proactive monitoring of data integration processes without direct querying of target Iceberg tables. When enabled by adding appropriate permissions on source and target processing role, CloudWatch metrics are automatically emitted to the `AWS/Glue/ZeroETL` namespace after completion of each table ingestion operation. You can setup alarms on your CloudWatch metrics to get notified when a particular Ingestion Job fails. + +### Metric Specifications + +**Ingestion Processing Metrics:** + + * `InsertCount`: Quantifies new records written to target Iceberg table + + * `UpdateCount`: Measures record modifications in target Iceberg table + + * `DeleteCount`: Tracks record removals from target Iceberg table @@ -113 +143,0 @@ Once an integration completes, you can see these Amazon Cloudwatch metrics gener -CloudWatch metrics namespace: "AWS/Glue/ZeroETL" @@ -115 +144,0 @@ CloudWatch metrics namespace: "AWS/Glue/ZeroETL" -Metrics dimensions: @@ -117 +145,0 @@ Metrics dimensions: - * `integrationArn` @@ -119 +147 @@ Metrics dimensions: - * `loadType` +**Operational Status Metrics:** @@ -121 +149 @@ Metrics dimensions: - * `tableName` + * `IngestionSucceeded`: Binary indicator (1) for successful ingestion completion @@ -122,0 +151 @@ Metrics dimensions: + * `IngestionFailed`: Binary indicator (1) for ingestion failure @@ -123,0 +153 @@ Metrics dimensions: + * `LastSyncTimestamp`: Temporal marker indicating the most recent successful source-to-target synchronization @@ -126 +155,0 @@ Metrics dimensions: -Metric names: @@ -128 +156,0 @@ Metric names: - * `InsertCount` \- number of records inserted in the target Iceberg table. @@ -130 +158 @@ Metric names: - * `UpdateCount` \- number of records updated in the target Iceberg table. +**Metric Dimensions:** @@ -132 +160 @@ Metric names: - * `DeleteCount` \- number of records deleted from the target Iceberg table. + * `integrationArn`: Unique identifier for the integration @@ -134 +162 @@ Metric names: - * `IngestionSucceeded` \- count 1, if the ingestion succeeded for the integration. + * `loadType`: Specifies the event processingType (SEED/CDC) @@ -136 +164 @@ Metric names: - * `IngestionFailed` \- count 1, if the ingestion failed for the integration. + * `tableName`: Target Iceberg table identifier @@ -138 +165,0 @@ Metric names: - * `LastSyncTimestamp` \- timestamp until which source has been synced to target. @@ -141,0 +169,23 @@ Metric names: +### IAM permissions required to enable CloudWatch Metrics + +When creating your integration, the following IAM permissions are needed by the source and target roles to enable CloudWatch Metrics for an integration. AWS Glue zero-ETL integrations use these permissions provided in the source and target roles to emit CloudWatch Metrics to customer accounts. + + + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "cloudwatch:PutMetricData" + ], + "Resource": "*", + "Condition": { + "StringEquals": { + "cloudwatch:namespace": "AWS/Glue/ZeroETL" + } + } + } + ] + } +