AWS sagemaker-unified-studio documentation change
Summary
Removed sections about automating lineage capture from AWS Glue/Redshift connections and Spark tools. Fixed typos and updated API references. Capitalized 'Catalog' in text.
Security assessment
Changes involve documentation restructuring, typo fixes, and removal of automation configuration details. No security vulnerabilities or security feature additions are mentioned in the diff.
Diff
diff --git a/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md b/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md index 0e23d40d0..6972d3c6b 100644 --- a/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md +++ b/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md @@ -5 +5 @@ -Types of lineage nodes in Amazon SageMaker Unified StudioKey attributes in lineage nodesVisualizing data lineageData lineage authorization in Amazon SageMaker Unified StudioData lineage sample experience in Amazon SageMaker Unified StudioAutomate lineage capture from Data connectionsAutomate lineage capture from toolsUsing Amazon SageMaker Unified Studio data lineage programmatically +Types of lineage nodes in Amazon SageMaker Unified StudioKey attributes in lineage nodesVisualizing data lineageData lineage authorization in Amazon SageMaker Unified StudioData lineage sample experience in Amazon SageMaker Unified StudioUsing Amazon SageMaker Unified Studio data lineage programmatically @@ -9 +9 @@ Types of lineage nodes in Amazon SageMaker Unified StudioKey attributes in linea -Data lineage in Amazon SageMaker Unified Studio is an OpenLineage-compatible feature that can help you to capture and visualize lineage events, from OpenLineage-enabled systems or through APIs, to trace data origins, track transformations, and view cross-organizational data consumption. It provides you with an overarching view into your data assets to see the origin of assets and their chain of connections. The lineage data includes information on the activities inside the Amazon SageMaker catalog, including information about the catalogued assets, the subscribers of those assets, and the activities that happen outside the business data catalog captured programmatically using the APIs. +Data lineage in Amazon SageMaker Unified Studio is an API-driven, OpenLineage-compatible feature that can help you to capture and visualize lineage events, from OpenLineage-enabled systems or through APIs, to trace data origins, track transformations, and view cross-organizational data consumption. It provides you with an overarching view into your data assets to see the origin of assets and their chain of connections. The lineage data includes information on the activities inside the Amazon SageMaker Catalog, including information about the catalogued assets, the subscribers of those assets, and the activities that happen outside the business data catalog captured programmatically using the APIs. @@ -78 +78 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Column search: when the default display for number of columns is 10. If there are more than 10 columns, pagination is activated to navigate to the rest of the columns. To quickly view a particular column, you can search on the dataset node that list just the searched column. + * Column search: when the default display for number of columns is 10. If there are more than 10 columns, pagination is activiated to navigate to the rest of the columns. To quickly view a particular column, you can search on the dataset node that list just the searched column. @@ -117 +117 @@ Complete the followng procedure to try the sample data lineage experience in Ama - 7. Choose **Start guided data lineage tour**. + 7. Choos **Start guided data lineage tour**. @@ -126,160 +125,0 @@ At this point, a tab that provides all the space of lineage information is displ -## Automate lineage capture from Data connections - -**Configure automated lineage capture for AWS Glue (Lakehouse) connections** - -As databases and tables are added to the Amazon SageMaker Unified Studio’s catalog, the lineage extraction can be automated from source for those assets using data source runs in Create Connection workflow. For every connection created, lineage is not automatically enabled. - -###### To enable lineage capture for an AWS Glue connection - - 1. Navigate to Amazon SageMaker Unified Studio using the URL from your admin and log in using your SSO or AWS credentials. - - 2. Choose **Select project** from the top navigation pane and select the project to which you want to add the data source. - - 3. Choose **Data sources** from the left navigation pane under Project catalog. - - 4. Choose the data source that you want to modify. - - 5. Expand the **Actions** menu, then choose **Edit data source** or click on the Data Source run name to view the details and go to **Data Source Definition** tab and choose **Edit** in **Connection** details. - - 6. Go to the connections and select **Import data lineage** checkbox to configure lineage capture from the source. - - 7. Make other changes to the data source fields as desired, then choose **Save**. - -**Limitations** - -The lineage collection in data source runs fetches information from table metadata to build lineage. AWS Glue crawler supports different types of sources for which lineage is captured, including Amazon S3, DynamoDB, Catalog, Delta Lake, Iceberg tables, and Hudi tables stored in Amazon S3. JDBC and DocumentDB or MongoDB are currently NOT supported as sources. - -If the number of tables is more than 100, the lineage run fails after 100 tables. Make sure that the AWS Glue crawler is not configured to bring in more that 100 tables in a run. - - - - -###### Note - -When enabled, the lineage runs asynchronously to capture metadata from the source and generate lineage events to be stored in SageMaker Catalog to be visualized from a particular asset. The status of lineage runs for the data source can be viewed along with data source run details. The lineage run is set up to run once daily. For the first run, after enabling the feature, the first pull is scheduled for ~5 minutes after and set for a daily run. You can configure specific time programmatically. - -**Configure automated lineage capture for Amazon Redshift connections** - -Capturing lineage from Amazon Redshift can be automated when the connection is added to an Amazon Redshift source in Amazon SageMaker Unified Studio’s Data explorer. Lineage capture can be automated for a connection at the data source configuration. For every connection created, lineage is not automatically enabled. - -###### To enable lineage capture for an Amazon Redshift connection - - 1. Navigate to Amazon SageMaker Unified Studio using the URL from your admin and log in using your SSO or AWS credentials. - - 2. Choose **Select project** from the top navigation pane and select the project to which you want to add the data source. - - 3. Choose **Data sources** from the left navigation pane under **Project catalog**. - - 4. Choose the data source that you want to modify. - - 5. Expand the **Actions** menu, then choose **Edit data****source** or click on the data source run name to view the details and go to Data Source Definition tab and select **Edit** in **Connection details**. - - 6. Go to the connections and select **Import data lineage** checkbox to configure lineage capture from the source. - - 7. Make other changes to the data source fields as desired, then choose **Save**. - - - - -###### Note - -When enabled, the lineage runs captures queries executed for a given database and generates lineage events to be stored in Amazon DataZone to be visualized from a particular asset. The lineage run for Amazon Redshift is set up for a daily run to pull from the Amazon Redshift system tables to derive lineage. For the first run, after enabling the feature, the first pull is scheduled for ~5 minutes after and set for a daily run. You can configure specific time programmatically. - -## Automate lineage capture from tools - -**Capture lineage for Spark executions in Visual ETL** - -When a new job is created in vETL in Amazon SageMaker Unified Studio, lineage is automatically enabled. When a Visual ETL flow is created, lineage capture for that ETL flow is automatically enabled when you choose **Save**. - -The following Spark configuration parameters are automatically added to the job being executed: - - - { - "--conf":"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener - --conf spark.openlineage.transport.type=amazon_datazone_api - --conf spark.openlineage.transport.domainId={DOMAIN_ID} - --conf spark.glue.accountId={ACCOUNT_ID} - --conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] - --conf spark.glue.JOB_NAME={JOB_NAME}" - } - - -The parameters are auto-configured and do not need any updates from the user. To understand the parameters in detail: - - * `spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener` \- OpenLineageSparkListener will be created and registered with Spark's listener bus - - * `spark.openlineage.transport.type=amazon_datazone_api` \- This is an OpenLineage specification to tell the OpenLineage Plugin to use DataZone API Transport to emit lineage events to DataZone’s PostLineageEvent API. For more information, see [https://openlineage.io/docs/integrations/spark/configuration/spark_conf/](https://openlineage.io/docs/integrations/spark/configuration/spark_conf/) - - * `spark.openlineage.transport.domainId={DOMAIN_ID}` \- This parameter establishes the domain to which the API transport will submit the lineage events to. - - * `spark.openlineage.facets.custom_environment_variables [AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;]` \- The following environment variables (`AWS_DEFAULT_REGION`, `GLUE_VERSION`, `GLUE_COMMAND_CRITERIA`, and `GLUE_PYTHON_VERSION`), which AWS Glue interactive session populates, will be added to the LineageEvent - - * `spark.glue.accountId=<ACCOUNT_ID>` \- Account Id of the Glue Data Catalog where the metadata resides. This account id is used to construct Glue ARN in lineage event. - - * `spark.glue.JOB_NAME` \- Job name of the lineage event. In vETL flow, the job name is configured automatically to be spark.glue.JOB_NAME: ${projectId}.${pathToNotebook} - - - - -**Capture lineage for Spark executions in Notebooks** - -Sessions in notebooks does not have a concept of a job. You can map the Spark executions to lineage events by generating a unique job name for the notebook. You can use the %%configure magic with the below parameters to enable lineage capture for Spark executions in the notebook. - - - %%configure --name project.spark -f - { - "--conf":"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openlineage.transport.type=amazon_datazone_api --conf spark.openlineage.transport.domainId={DOMAIN_ID} --conf spark.glue.accountId={ACCOUNT_ID} --conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION; --conf spark.glue.JOB_NAME={JOB_NAME}" - } - - -The following are the parameter details: - - * `spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener` \- OpenLineageSparkListener will be created and registered with Spark's listener bus - - * `spark.openlineage.transport.type=amazon_datazone_api` \- This is an OpenLineage specification to tell the OpenLineage Plugin to use DataZone API Transport to emit lineage events to DataZone’s PostLineageEvent API. For more information, see [https://openlineage.io/docs/integrations/spark/configuration/spark_conf](https://openlineage.io/docs/integrations/spark/configuration/spark_conf) - - * `spark.openlineage.transport.domainId={DOMAIN_ID}` \- This parameter establishes the domain to which the API transport will submit the lineage events to. - - * `spark.openlineage.facets.custom_environment_variables [AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;]` \- The following environment variables (AWS_DEFAULT_REGION , GLUE_VERSION , GLUE_COMMAND_CRITERIA, and GLUE_PYTHON_VERSION), which Glue interactive session populates, will be added to the LineageEvent - - * `spark.glue.accountId=<ACCOUNT_ID>` \- Account Id of the Glue Data Catalog where the metadata resides. This account id is used to construct Glue ARN in lineage event. - - * `spark.glue.JOB_NAME` \- Job name of the lineage event. In vETL flow, the job name is configured automatically to be spark.glue.JOB_NAME: ${projectId}.${pathToNotebook} - - - - -**Capture lineage EMR-S Spark executions from Notebooks** - -EMR v7.5 and greater with Spark engine has the necessary OpenLineage libraries built in. They need to be added to the spark submit properties in order to be used especially if AWS Glue is being used as the Hive metastore. The rest of the spark submit properties are similar to those used in AWS Glue jobs. Be sure to replace the {Domain ID} with your specific Amazon DataZone or Amazon SageMaker Unified Studio domain and to replace the {Account ID} with the account id where the EMR job is run. - - - { - "--conf spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener - --conf spark.openlineage.transport.type=amazon_datazone_api - --conf spark.openlineage.transport.domainId={Domain ID} - --conf spark.jars=/usr/share/aws/datazone-openlineage-spark/lib/DataZoneOpenLineageSpark-1.0.jar - --conf spark.glue.accountId={Account ID}" - } - - - * Replace DATAZONE_DOMAIN_ID and ACCOUNT_ID with valid values - - * Amazon DataZone VPCE is deployed to EMR-S VPC - - * The JOB_NAME is the Spark application name that is automaticaly set - - - - -**Limitations** - - * OpenLineage libraries for Spark are built into AWS Glue v5.0+ for Spark DataFrames only. Does not support Dynamic DataFrames. - - * OpenLineage libraries for Spark are built into Amazon EMR v7.5+ and only for EMR-S. Lineage is not supported in in EMR on EKS and EMR on EC2. - - * LineageEvent has a size limit of 300KB. - - - - @@ -290,2 +129,0 @@ To use the data lineage functionality in Amazon SageMaker Unified Studio, you ca - * [GetLineageEvent](https://docs.aws.amazon.com/datazone/latest/APIReference/API_GetLineageEvent.html) - @@ -294,2 +131,0 @@ To use the data lineage functionality in Amazon SageMaker Unified Studio, you ca - * [ListLineageEvents](https://docs.aws.amazon.com/datazone/latest/APIReference/API_ListLineageEvents.html) -