AWS sagemaker-unified-studio documentation change
Summary
Expanded EMR Spark lineage support, added column lineage enforcement, and updated configuration requirements
Security assessment
Changes involve feature expansion (adding EMR on EC2/EKS support) and configuration updates. The enforced column lineage (datasetLineageEnabled=True) improves data governance but isn't a security feature. No vulnerability remediation evident.
Diff
diff --git a/sagemaker-unified-studio/latest/userguide/datazone-data-lineage-automate-capture-from-tools.md b/sagemaker-unified-studio/latest/userguide/datazone-data-lineage-automate-capture-from-tools.md index 63630d70c..506ed40d4 100644 --- a//sagemaker-unified-studio/latest/userguide/datazone-data-lineage-automate-capture-from-tools.md +++ b//sagemaker-unified-studio/latest/userguide/datazone-data-lineage-automate-capture-from-tools.md @@ -5 +5 @@ -Capture lineage for Spark executions in Visual ETLCapture lineage for AWS Glue Spark executions in NotebooksCapture lineage EMR-S Spark executions from Notebooks +Capture lineage for Spark executions in Visual ETLCapture lineage for AWS Glue Spark executions in NotebooksCapture lineage from EMR Spark executions @@ -15 +15 @@ Capture lineage for Spark executions in Visual ETLCapture lineage for AWS Glue S - * Capture lineage EMR-S Spark executions from Notebooks + * Capture lineage from EMR Spark executions @@ -35 +35 @@ The following Spark configuration parameters are automatically added to the job - --conf spark.openlineage.columnLineage.datasetLineageEnabled={True|False} + --conf spark.openlineage.columnLineage.datasetLineageEnabled=True @@ -61,4 +60,0 @@ The parameters are auto-configured and do not need any updates from the user. To - * OpenLineage libraries for Spark are built into Amazon EMR v7.5+ and only for EMR-S. - - * Capturing lineage from Spark jobs executed on EMR on EKS and EMR on EC2 are not automated but can be done by manual configuration. - @@ -79 +75 @@ Note: for AWS Glue Spark executions in notebooks, lineage capture is automated w - "--conf":"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openlineage.transport.type=amazon_datazone_api --conf spark.openlineage.transport.domainId={DOMAIN_ID} --conf spark.glue.accountId={ACCOUNT_ID} --conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] --conf spark.glue.JOB_NAME={JOB_NAME}" + "--conf":"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openlineage.transport.type=amazon_datazone_api --conf spark.openlineage.transport.domainId={DOMAIN_ID} --conf spark.glue.accountId={ACCOUNT_ID} --conf spark.openlineage.columnLineage.datasetLineageEnabled=True --conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] --conf spark.glue.JOB_NAME={JOB_NAME}" @@ -131 +127 @@ Here are these parameters and what they configure, in detail: -## Capture lineage EMR-S Spark executions from Notebooks +## Capture lineage from EMR Spark executions @@ -133 +129 @@ Here are these parameters and what they configure, in detail: -EMR v7.5 and greater with Spark engine has the necessary OpenLineage libraries built in. They need to be added to the spark submit properties in order to be used especially if AWS Glue is being used as the Hive metastore. The rest of the spark submit properties are similar to those used in AWS Glue jobs. Be sure to replace the {Domain ID} with your specific Amazon DataZone or Amazon SageMaker Unified Studio domain and to replace the {Account ID} with the account id where the EMR job is run. +EMR with Spark engine has the necessary OpenLineage libraries built in. You need to pass the following spark parameters. Be sure to replace the {Domain ID} with your specific Amazon DataZone or Amazon SageMaker Unified Studio domain and to replace the {Account ID} with the account id where the EMR job is run. @@ -142,2 +138,3 @@ EMR v7.5 and greater with Spark engine has the necessary OpenLineage libraries b - "spark.glue.accountId":"{ACCOUNT_ID}", - "spark.jars":"/usr/share/aws/datazone-openlineage-spark/lib/DataZoneOpenLineageSpark-1.0.jar" + "spark.glue.accountId":"{ACCOUNT_ID}", // needed if AWS Glue is being used as the Hive metastore + "spark.openlineage.columnLineage.datasetLineageEnabled":"True", + "spark.jars":"/usr/share/aws/datazone-openlineage-spark/lib/DataZoneOpenLineageSpark-1.0.jar" // Only needed incase of EMR-S @@ -147,0 +145,8 @@ EMR v7.5 and greater with Spark engine has the necessary OpenLineage libraries b + * Lineage is supported for the following EMR versions: + + * EMR-S: 7.5+ + + * EMR on EC2: 7.11+ + + * EMR on EKS: 7.12+ + @@ -152 +157 @@ EMR v7.5 and greater with Spark engine has the necessary OpenLineage libraries b - * Amazon SageMaker Unified Studio VPC endpoint is deployed to EMR serverless VPC endpoint + * Amazon SageMaker Unified Studio VPC endpoint is deployed to EMR VPC endpoint @@ -165 +170 @@ Automate lineage capture from data connections -Publishing data lineage programmatically +Permissions required for data lineage