AWS sagemaker-unified-studio documentation change
Summary
Expanded troubleshooting guide for data lineage visibility issues. Added new sections for EMR-S/EC2/EKS lineage troubleshooting, detailed limitations for Redshift lineage, enhanced Glue troubleshooting steps, and added CloudWatch log queries for diagnostics.
Security assessment
The changes focus on operational troubleshooting and lineage visibility limitations. While they mention security configurations like KMS permissions and VPC endpoints, these are standard operational guidance rather than addressing specific vulnerabilities. There's no evidence of patching security flaws or incident response.
Diff
diff --git a/sagemaker-unified-studio/latest/userguide/datazone-lineage-troubleshooting.md b/sagemaker-unified-studio/latest/userguide/datazone-lineage-troubleshooting.md index 9297872f7..b5b5c090a 100644 --- a//sagemaker-unified-studio/latest/userguide/datazone-lineage-troubleshooting.md +++ b//sagemaker-unified-studio/latest/userguide/datazone-lineage-troubleshooting.md @@ -5 +5 @@ -Not seeing lineage graph for events published programmaticallyNot seeing lineage for assets even though importLineage is shown as true in AWS Glue datasourceNot seeing lineage for assets even though importLineage is shown as true in Amazon Redshift datasourceNot seeing lineage for lineage events published from AWS Glue ETL jobs/vETL/Notebooks +Not seeing lineage graph for events published programmaticallyNot seeing lineage for assets even though importLineage is shown as true in AWS Glue datasourceNot seeing lineage for assets even though importLineage is shown as true in Amazon Redshift datasourceTroubleshooting lineage for lineage events published from AWS Glue ETL jobs/vETL/NotebooksTroubleshooting lineage for lineage events published from EMR-S/EC2/EKS @@ -19 +19,3 @@ This comprehensive troubleshooting guide helps you resolve common data lineage v - * Not seeing lineage for lineage events published from AWS Glue ETL jobs/vETL/Notebooks + * Troubleshooting lineage for lineage events published from AWS Glue ETL jobs/vETL/Notebooks + + * Troubleshooting lineage for lineage events published from EMR-S/EC2/EKS @@ -167 +169,24 @@ Lineage on Amazon Redshift tables is captured by retrieving user queries perform -Following are the troubleshooting steps if you don't see the lineage even after enabling it: +**Lineage is not supported in the following cases:** + + * External Tables + + * Unload / Copy + + * Merge / Update + + * Queries that produce Lineage Events larger than 16MB + + * Datashares + + * ColumnLineage limitations: + + * Column Lineage is not supported for queries not containing specific columns such as (select * from tableA) + + * Column Lineage is not supported for queries involving temp tables + + * Any limitations pertaining to [OpenLineageSqlParser](https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md) results in failure to process some queries + + + + +**Troubleshooting steps:** @@ -228 +253 @@ The response appears as follows: -Lineage is not supported in the following cases: +## Troubleshooting lineage for lineage events published from AWS Glue ETL jobs/vETL/Notebooks @@ -230 +255 @@ Lineage is not supported in the following cases: - * External Tables +**Limitations:** @@ -232 +257 @@ Lineage is not supported in the following cases: - * Unload / Copy + * OpenLineage libraries for Spark are built into AWS Glue v5.0+ for Spark DataFrames only. Does not support Glue DynamicFrames. @@ -234 +259 @@ Lineage is not supported in the following cases: - * Merge / Update + * Lineage capture for Spark jobs with fine-grained permission mode are not supported. @@ -236 +261 @@ Lineage is not supported in the following cases: - * Queries that produce Lineage Events larger than 16MB + * Lineage event has a size limit of 300KB. @@ -238 +262,0 @@ Lineage is not supported in the following cases: - * Datashares @@ -240 +263,0 @@ Lineage is not supported in the following cases: - * Any limitations pertaining to [OpenLineageSqlParser](https://github.com/OpenLineage/OpenLineage/blob/main/integration/sql/README.md) results in failure to process some queries @@ -242,0 +266 @@ Lineage is not supported in the following cases: +**Common Issues:** @@ -243,0 +268 @@ Lineage is not supported in the following cases: + * If the AWS Glue ETL is in VPC, make sure the Amazon DataZone VPC endpoint is deployed in that VPC. @@ -245 +270 @@ Lineage is not supported in the following cases: -## Not seeing lineage for lineage events published from AWS Glue ETL jobs/vETL/Notebooks + * In case your domain is using a CMK, make sure that the AWS Glue execution role has the appropriate KMS permissions. CMK can be found via <https://docs.aws.amazon.com/datazone/latest/APIReference/API_GetDomain.html> @@ -247 +272 @@ Lineage is not supported in the following cases: -**Limitations:** + * Failed to publish Lineage Event because payload is greater than 300 kb: @@ -249 +274 @@ Lineage is not supported in the following cases: - * OpenLineage libraries for Spark are built into AWS Glue v5.0+ for Spark DataFrames only. Does not support Glue DynamicFrames. + * Add the following to Spark conf: @@ -251 +276 @@ Lineage is not supported in the following cases: - * OpenLineage libraries for Spark are built into Amazon EMR v7.5+ and only for EMR-S. +` "spark.openlineage.columnLineage.datasetLineageEnabled": "True" ` @@ -253 +278 @@ Lineage is not supported in the following cases: - * Capturing lineage from Spark jobs executed on EMR on EKS and EMR on EC2 are not automated but can be done by manual configuration. + * **Important Note:** @@ -255 +280 @@ Lineage is not supported in the following cases: - * Lineage capture for Spark jobs with fine-grained permission mode are not supported. + * Column lineage typically constitutes a significant portion of the payload and enabling this will efficiently generate the column lineage info. @@ -257 +282 @@ Lineage is not supported in the following cases: - * If you are trying EMR-S outside of Amazon SageMaker Unified Studio, Amazon DataZone VPC endpoint needs to be deployed for EMR-S VPC. + * Disabling it can help reduce payload size and avoid validation exceptions. @@ -259 +284 @@ Lineage is not supported in the following cases: - * Lineage event has a size limit of 300KB. + * Cross account lineage event submission: @@ -260,0 +286 @@ Lineage is not supported in the following cases: + * Follow <https://docs.aws.amazon.com/datazone/latest/userguide/working-with-associated-accounts.html> to set up account association @@ -261,0 +288 @@ Lineage is not supported in the following cases: + * Ensure that RAM policy is using the latest policy @@ -262,0 +290 @@ Lineage is not supported in the following cases: + * When the same DataFrame is written to multiple destinations or formats in sequence, Lineage SparkListener may only capture the lineage for the first write operation: @@ -264 +292,6 @@ Lineage is not supported in the following cases: - 1. Lineage graph can only be visualized if at least one node of the graph is an asset. Therefore, create assets for any of the datasets (such as tables) involved in the job and then attempt to visualize lineage on the asset. + * For optimization purposes, Spark's internals may reuse execution plans definition for consecutive write operations on the same DataFrame. This can lead to only capturing first lineage event. + + + + +**Troubleshooting steps:** @@ -266 +299 @@ Lineage is not supported in the following cases: - 2. **Troubleshooting steps:** + 1. Lineage graph can only be visualized if at least one node of the graph is an asset. Therefore, create assets for any of the datasets (such as tables) involved in the job and then attempt to visualize lineage on the asset. @@ -268 +301 @@ Lineage is not supported in the following cases: - 1. First, invoke [ListLineageEvents](https://docs.aws.amazon.com/datazone/latest/APIReference/API_ListLineageEvents.html) to see if the lineage events were submitted (refer to the linked doc to pass filters). + 2. First, invoke [ListLineageEvents](https://docs.aws.amazon.com/datazone/latest/APIReference/API_ListLineageEvents.html) to see if the lineage events were submitted (refer to the linked doc to pass filters). @@ -270 +303 @@ Lineage is not supported in the following cases: - 2. If no events are submitted, check AWS CloudWatch logs to see if any exceptions are thrown from the Amazon DataZone Lineage lib: + 3. If no events are submitted, check AWS CloudWatch logs to see if any exceptions are thrown from the Amazon DataZone Lineage lib: @@ -276 +309 @@ Lineage is not supported in the following cases: - * Following is the AWS CloudWatch log insights query: + * Following is the AWS CloudWatch log insights query to check for exceptions: @@ -283 +316,10 @@ Lineage is not supported in the following cases: - 3. Enable Spark UI to see if Spark Logical Plans and Spark Confs are generated properly: <https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html#monitor-spark-ui-jobs-cli> + * Following is the AWS CloudWatch log insights query to inspect generated events: + +Pass `--conf spark.log.level=DEBUG` while submitting the spark job. + + fields @timestamp, @message + | filter @message like /Emitting lineage completed successfully/ + | sort @timestamp desc + + + 4. Enable Spark UI to see if Spark Logical Plans and Spark Confs are generated properly: <https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui-jobs.html#monitor-spark-ui-jobs-cli> @@ -292 +334 @@ Under Environment, all the spark configurations passed could be found. Verify th - 3. **Common Issues:** + 5. After verifying lineage events are successfully processed in AWS CloudWatch logs, follow the steps in Not seeing lineage graph for events published programmatically to troubleshoot. @@ -294 +336 @@ Under Environment, all the spark configurations passed could be found. Verify th - * If the AWS Glue ETL is in VPC, make sure the Amazon DataZone VPC endpoint is deployed in that VPC. + 6. If you are still unable to see lineage and it doesn't fall under the limitations category, reach out to AWS support by providing: @@ -296 +338 @@ Under Environment, all the spark configurations passed could be found. Verify th - * In case your domain is using a CMK, make sure that the AWS Glue execution role has the appropriate KMS permissions. CMK can be found via <https://docs.aws.amazon.com/datazone/latest/APIReference/API_GetDomain.html> + * Spark config parameters @@ -298 +340 @@ Under Environment, all the spark configurations passed could be found. Verify th - * Failed to publish Lineage Event because payload is greater than 300 kb: + * Lineage event from GetLineageEvent response for successfully processed events to which lineage isn't visible @@ -300 +342 @@ Under Environment, all the spark configurations passed could be found. Verify th - * Add the following to Spark conf: + * GetAsset response @@ -302 +343,0 @@ Under Environment, all the spark configurations passed could be found. Verify th - "spark.openlineage.columnLineage.datasetLineageEnabled": "true" @@ -305 +345,0 @@ Under Environment, all the spark configurations passed could be found. Verify th - * **Important Note:** @@ -307 +347 @@ Under Environment, all the spark configurations passed could be found. Verify th - * Column lineage typically constitutes a significant portion of the payload and enabling this will efficiently generate the column lineage info. +## Troubleshooting lineage for lineage events published from EMR-S/EC2/EKS @@ -309 +349,28 @@ Under Environment, all the spark configurations passed could be found. Verify th - * Disabling it can help reduce payload size and avoid validation exceptions. +**Notes:** + + * Lineage is supported from following versions of EMR: + + * EMR-S: 7.5+ + + * EMR-EC2: 7.11+ + + * EMR-EKS: 7.12+ + + * Lineage capture for Spark jobs with fine-grained permission mode are not supported. + + * If you are trying EMR outside of Amazon SageMaker Unified Studio, Amazon DataZone VPC endpoint needs to be deployed for EMR VPC. + + * Lineage event has a size limit of 300KB. + + + + +**Common Issues:** + + * In case your domain is using a CMK, make sure that the job's execution role has the appropriate KMS permissions. CMK can be found via <https://docs.aws.amazon.com/datazone/latest/APIReference/API_GetDomain.html> + + * Failed to publish Lineage Event because payload is greater than 300 kb: + + * Add the following to Spark conf which efficiently generates the event payload: + +` "spark.openlineage.columnLineage.datasetLineageEnabled": "true" ` @@ -320,0 +388,45 @@ Under Environment, all the spark configurations passed could be found. Verify th + + + +**Troubleshooting steps:** + + 1. Lineage graph can only be visualized if at least one node of the graph is an asset. Therefore, create assets for any of the datasets (such as tables) involved in the job and then attempt to visualize lineage on the asset. + + 2. First, invoke [ListLineageEvents](https://docs.aws.amazon.com/datazone/latest/APIReference/API_ListLineageEvents.html) to see if the lineage events were submitted (refer to the linked doc to pass filters). + + 3. If no events are submitted, check AWS CloudWatch logs to see if any exceptions are thrown from the Amazon DataZone Lineage lib: + + * **EC2:** + + * You can enable provide the CloudWatch log group or log destination for S3 path at the time of creating EC2 cluster. Refer <https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-debugging.html> + + * You will see logs in stderr file within cluster-id/containers/application*/ folder. + + * **EKS:** + + * You need to provide the CloudWatch log group or log destination for S3 path while submitting spark job. Refer <https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/getting-started.html> + + * You will see logs in stderr file of spark_driver within virtual-cluster-id/jobs/job-id/containers/* folder. + + * **EMR-S:**