AWS sagemaker-unified-studio documentation change
Summary
Major restructuring of data lineage documentation with added OpenLineage integration details, authorization requirements, and expanded configuration examples. Removed node type descriptions and added sections about programmatic publishing, sourceIdentifier importance, and asset linking.
Security assessment
The changes add documentation about IAM role requirements for lineage event publishing (PostLineageEvent API permissions) and clarify read access controls. While this describes security-related configurations, there's no evidence of addressing a specific vulnerability. The authorization section explains standard access controls rather than patching a security issue.
Diff
diff --git a/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md b/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md index 0e23d40d0..35f3c47e8 100644 --- a//sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md +++ b//sagemaker-unified-studio/latest/userguide/datazone-data-lineage.md @@ -5 +5 @@ -Types of lineage nodes in Amazon SageMaker Unified StudioKey attributes in lineage nodesVisualizing data lineageData lineage authorization in Amazon SageMaker Unified StudioData lineage sample experience in Amazon SageMaker Unified StudioAutomate lineage capture from Data connectionsAutomate lineage capture from toolsUsing Amazon SageMaker Unified Studio data lineage programmatically +What is OpenLineage?Lineage support in Amazon SageMaker Unified StudioVisualizing data lineageData lineage sample experience in Amazon SageMaker Unified StudioData lineage authorization in Amazon SageMaker Unified StudioAutomate lineage capture from data connectionsAutomate lineage capture from toolsPublishing Amazon SageMaker Unified Studio data lineage programmaticallyThe importance of the sourceIdentifier attribute to lineage nodesLinking dataset lineage nodes with assets imported into Amazon SageMaker Unified Studio @@ -11 +11 @@ Data lineage in Amazon SageMaker Unified Studio is an OpenLineage-compatible fea -Using Amazon SageMaker Unified Studio's OpenLineage-compatible APIs, domain administrators and data producers can capture and store lineage events beyond what is available in Amazon SageMaker Unified Studio, including transformations in Amazon S3, AWS Glue, and other services. This provides a comprehensive view for the data consumers and helps them gain confidence of the asset's origin, while data producers can assess the impact of changes to an asset by understanding its usage. Additionally, Amazon SageMaker Unified Studio versions lineage with each event, enabling users to visualize lineage at any point in time or compare transformations across an asset's or job's history. This historical lineage provides a deeper understanding of how data has evolved, essential for troubleshooting, auditing, and ensuring the integrity of data assets. +###### Topics @@ -13 +13 @@ Using Amazon SageMaker Unified Studio's OpenLineage-compatible APIs, domain admi -With data lineage, you can accomplish the following in Amazon SageMaker Unified Studio: + * What is OpenLineage? @@ -15 +15 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified - * Understand the provenance of data: knowing where the data originated fosters trust in data by providing you with a clear understanding of its origins, dependencies, and transformations. This transparency helps in making confident data-driven decisions. + * Lineage support in Amazon SageMaker Unified Studio @@ -17 +17 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified - * Understand the impact of changes to data pipelines: when changes are made to data pipelines, lineage can be used to identify all of the downstream consumers that are to be affected. This helps to ensure that changes are made without disrupting critical data flows. + * Visualizing data lineage @@ -19 +19 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified - * Identify the root cause of data quality issues: if a data quality issue is detected in a downstream report, lineage, especially column-level lineage, can be used to trace the data back (at a column level) to identify the issue back to its source. This can help data engineers to identify and fix the problem. + * Data lineage sample experience in Amazon SageMaker Unified Studio @@ -21 +21 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified - * Improve data governance and compliance: column-level lineage can be used to demonstrate compliance with data governance and privacy regulations. For example, column-level lineage can be used to show where sensitive data (such as PII) is stored and how it is processed in downstream activities. + * Data lineage authorization in Amazon SageMaker Unified Studio @@ -22,0 +23 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified + * Automate lineage capture from data connections @@ -23,0 +25 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified + * Automate lineage capture from tools @@ -24,0 +27 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified + * Publishing Amazon SageMaker Unified Studio data lineage programmatically @@ -26 +29 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified -## Types of lineage nodes in Amazon SageMaker Unified Studio + * The importance of the sourceIdentifier attribute to lineage nodes @@ -28 +31 @@ With data lineage, you can accomplish the following in Amazon SageMaker Unified -in Amazon SageMaker Unified Studio, data lineage information is presented in nodes that represent tables and views. Depending on the context of the project, the producers are able to view both inventory and published assets, whereas consumers can only view the published assets. When you first open the lineage tab in the asset details page, the catalogued dataset node is the starting point for navigating upstream or downstream through the lineage nodes of your lineage graph. + * Linking dataset lineage nodes with assets imported into Amazon SageMaker Unified Studio @@ -30 +32,0 @@ in Amazon SageMaker Unified Studio, data lineage information is presented in nod -The following are the types of data lineage nodes that are supported in Amazon SageMaker Unified Studio: @@ -32 +33,0 @@ The following are the types of data lineage nodes that are supported in Amazon S - * **Dataset node** \- this node type includes data lineage information about a specific data asset. @@ -34 +34,0 @@ The following are the types of data lineage nodes that are supported in Amazon S - * Dataset nodes that include information about AWS Glue or Amazon Redshift assets published in the Amazon SageMaker Unified Studio catalog are auto-generated and include a corresponding AWS Glue or Amazon Redshift icon within the node. @@ -36 +36 @@ The following are the types of data lineage nodes that are supported in Amazon S - * Dataset nodes that include information about assets that are not published in the Amazon SageMaker Unified Studio catalog, are created manually by domain administrators (producers) and are represented by a default custom asset icon within the node. +## What is OpenLineage? @@ -38 +38 @@ The following are the types of data lineage nodes that are supported in Amazon S - * **Job (run) node** \- this node type displays the details of the job, including the latest run of a particular job and run details. This node also captures multiple runs of the job and can be viewed in the **History** tab of the node details. You can view node details by choosing the node icon. +[OpenLineage](https://openlineage.io/) is an open platform for collection and analysis of data lineage. It tracks metadata about datasets, jobs, and runs, giving users the information required to identify the root cause of complex issues and understand the impact of changes. It is an Open Standard for lineage metadata collection designed to record metadata for a job in execution. @@ -39,0 +40 @@ The following are the types of data lineage nodes that are supported in Amazon S +The standard defines a generic model of dataset, job, and run entities uniquely identified using consistent naming strategies. The dataset and job entities are identified by combination of 'namespace' and 'name' attributes whereas run is identified by runId. The entities can be enriched with user-defined metadata via facets (similar to metadata forms in Amazon SageMaker Unified Studio). @@ -40,0 +42 @@ The following are the types of data lineage nodes that are supported in Amazon S +OpenLineage supports three types of events: RunEvent, DatasetEvent and JobEvent. @@ -41,0 +44 @@ The following are the types of data lineage nodes that are supported in Amazon S + * RunEvent: this event is generated as a result of job-run execution. It contains details of the run, the job it belongs to, input datasets that run consumes and output datasets the run produces. Reference for samples run events. Currently, Amazon SageMaker Unified Studio only supports RunEvents. @@ -43 +46 @@ The following are the types of data lineage nodes that are supported in Amazon S -## Key attributes in lineage nodes + * DatasetEvent: this event represents the changes in dataset (like any static updates on the dataset) @@ -45 +48 @@ The following are the types of data lineage nodes that are supported in Amazon S -The `sourceIdentifier` attribute in a lineage node represents the events happening on a dataset. The `sourceIdentifier` of the lineage node is the identifier of the dataset (table/view etc). It’s used for uniqueness enforcement on the lineage nodes. For example, there can’t be two lineage nodes with same `sourceIdentifier`. The following are examples of `sourceIdentifier` values for different types of nodes: + * JobEvent: this event represents the changes in job configuration/details @@ -47 +49,0 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For dataset node with respective dataset type: @@ -49 +50,0 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * Asset: amazon.datazone.asset/<assetId> @@ -51 +51,0 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * Listing (published asset): amazon.datazone.listing/<listingId> @@ -53 +53 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * AWS Glue table: arn:aws:glue:<region>:<account-id>:table/<database>/<table-name> +In the current release of Amazon SageMaker Unified Studio, OpenLineage 1.22.0+ versions are supported. @@ -55 +55 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * Amazon Redshift table/view: arn:aws:<redshift/redshift-serverless>:<region>:<account-id>:<table-type(table/view etc)>/<clusterIdentifier/workgroupName>/<database>/<schema>/<table-name> +## Lineage support in Amazon SageMaker Unified Studio @@ -57 +57 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For any other type of dataset nodes imported using open-lineage run events, <namespace>/<name> of the input/output dataset is used as `sourceIdentifier` of the node. +In Amazon SageMaker Unified Studio, domain administrators or data users can configure lineage in projects while setting up connections for data lake and data warehouse sources to ensure the data source runs created from those resources are enabled for automatic lineage capture. Data lineage is automatically captured from data sources, such as AWS Glue and Amazon Redshift, as well as from tools, such as Visual ETL and notebooks, as executions create, update, or transform data. Additionally, Amazon SageMaker Unified Studio captures the movement of data within the catalog as producers bring their asset into inventory, and publish them, as well as when consumers subscribe and get access, to indicate who the subscribing projects are for a given asset. With this automation, different stages of an asset in the catalog are captured including when schema changes are detected. @@ -59 +59 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For jobs: +Using Amazon SageMaker Unified Studio's OpenLineage-compatible APIs, domain administrators and data producers can capture and store lineage events beyond what is available in Amazon SageMaker Unified Studio, including transformations in Amazon S3, AWS Glue, and other services. This provides a comprehensive view for the data consumers and helps them gain confidence of the asset's origin, while data producers can assess the impact of changes to an asset by understanding its usage. Additionally, Amazon SageMaker Unified Studio versions lineage with each event, enabling users to visualize lineage at any point in time or compare transformations across an asset's or job's history. This historical lineage provides a deeper understanding of how data has evolved, essential for troubleshooting, auditing, and ensuring the integrity of data assets. @@ -61 +61 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For job nodes imported using open-lineage run events, <jobs_namespace>.<job_name> is used as sourceIdentifier. +With data lineage, you can accomplish the following in Amazon SageMaker Unified Studio: @@ -63 +63,7 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For job runs: + * Understand the provenance of data: knowing where the data originated fosters trust in data by providing you with a clear understanding of its origins, dependencies, and transformations. This transparency helps in making confident data-driven decisions. + + * Understand the impact of changes to data pipelines: when changes are made to data pipelines, lineage can be used to identify all of the downstream consumers that are to be affected. This helps to ensure that changes are made without disrupting critical data flows. + + * Identify the root cause of data quality issues: if a data quality issue is detected in a downstream report, lineage, especially column-level lineage, can be used to trace the data back (at a column level) to identify the issue back to its source. This can help data engineers to identify and fix the problem. + + * Improve data governance and compliance: column-level lineage can be used to demonstrate compliance with data governance and privacy regulations. For example, column-level lineage can be used to show where sensitive data (such as PII) is stored and how it is processed in downstream activities. @@ -65 +70,0 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni - * For job run nodes imported using open-lineage run events, <jobs_namespace>.<job_name>/<run_id> is used as sourceIdentifier. @@ -68,0 +74 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni +**OpenLineage custom transport to send lineage events to SageMaker** @@ -70 +76 @@ The `sourceIdentifier` attribute in a lineage node represents the events happeni -For assets created using `createAsset` API, the `sourceIdentifier` must be updated using `createAssetRevision` API to enable mapping the asset to upstream resources. +OpenLineage events which contain metadata about data pipelines, jobs, and runs, are typically sent to backend for storage and analysis. The transport mechanism handles this transmission. As an extension of the OpenLineage project, a custom transport is available to send lineage events directly Amazon SageMaker Unified Studio's endpoint. The custom transport was merged into OpenLineage version 1.33.0 ([https://openlineage.io/docs/releases/1_33_0/](https://openlineage.io/docs/releases/1_33_0/)). This allows the use of OpenLineage plugins with the transport to send lineage events collected directly to Amazon SageMaker Unified Studio. @@ -74 +80 @@ For assets created using `createAsset` API, the `sourceIdentifier` must be updat -Amazon SageMaker Unified Studio’s asset details page provides a graphical representation of data lineage, making it easier to visualize data relationships upstream or downstream. The asset details page provides the following capabilities to navigate the graph: +In Amazon SageMaker Unified Studio, nodes in the lineage graph contain lineage information while edges represent upstream/downstream directions of data propagation. The lineage information is present in metadata forms attached to the lineage node. Amazon SageMaker Unified Studio defines three types of lineage nodes: @@ -76 +82 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Column-level lineage: expand column-level lineage when available in dataset nodes. This automatically shows relationships with upstream or downstream dataset nodes if source column information is available. + * Dataset node - this node includes data lineage information about a specific dataset. @@ -78 +84 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Column search: when the default display for number of columns is 10. If there are more than 10 columns, pagination is activated to navigate to the rest of the columns. To quickly view a particular column, you can search on the dataset node that list just the searched column. + * Dataset refers to any object such as table, view, Amazon S3 file, Amazon S3 bucket, etc. It also refers to the assets in Amazon SageMaker Unified Studio's inventory and catalog, and subscribed tables/views. @@ -80 +86 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * View dataset nodes only: if you want to toggle to view only dataset lineage nodes and filter out the job nodes, you can choose the Open view control icon on the top left of the graph viewer and toggle the **Display dataset nodes only** option. This will remove all the job nodes from the graph and lets you navigate just the dataset nodes. Note that when the view only dataset nodes is turned on, the graph cannot be expanded upstream or downstream. + * Each version of the dataset node represents an event happening on the dataset at that timestamp. The history tab on the dataset node shows all dataset versions. @@ -82 +88 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Details pane: Each lineage node has details captured and displayed when selected. + * Job node - this node includes job details such as type of the job (query, etl etc), processing type (batch, streaming) etc job-type (query, etl etc), processing-type etc. @@ -84 +90 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Dataset node has a detail pane to display all the details captured for that node for a given timestamp. Every dataset node has 3 tabs, namely: Lineage info, Schema, and History tab. The history tab lists the different versions of lineage event captured for that node. All details captured from API are displayed using metadata forms or a JSON viewer. + * JobRun node - this node represents the job run details such as the job it belongs to, status, start/end timestamps etc. Amazon SageMaker Unified Studio's lineage graph shows a combined for job and job-run which shows job details and latest run details along with a history of previous job-runs. @@ -86 +91,0 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Job node has a detail pane to display job details with tabs, namely: Job info, and History. The details pane also captures query or expressions captured as part of the job run. The history tab lists the different versions of job run event captured for that job. All details captured from API are displayed using metadata forms or a JSON viewer. @@ -88 +92,0 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr - * Version tabs: all lineage nodes in Amazon SageMaker Unified Studio data lineage have versioning. For every dataset node or job node, the versions are captured as history and that enables you to navigate between the different versions to identify what has changed overtime. Each version opens a new tab in the lineage page to help compare or contrast. @@ -90,0 +95 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr +Lineage graph can be visualized with base node as an asset. In the SageMaker Unified Studio, search for the assets, open any asset and you can see lineage on the Asset Details page. @@ -91,0 +97,23 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr +Here is the sample lineage graph for a user who is a data producer: + + + +Here is the sample lineage graph for a user who is a data consumer: + + + +The asset details page provides the following capabilities to navigate the graph: + + * Column-level lineage: expand column-level lineage when available in dataset nodes. This automatically shows relationships with upstream or downstream dataset nodes if source column information is available. + + * Column search: the default display for number of columns is 10. If there are more than 10 columns, pagination is activated to navigate to the rest of the columns. To quickly view a particular column, you can search on the dataset node that list just the searched column. + + * View dataset nodes only: if you want to toggle to view only dataset lineage nodes and filter out the job nodes, you can choose the **Open view control** icon on the top left of the graph viewer and toggle the **Display dataset nodes only** option. This will remove all the job nodes from the graph and lets you navigate just the dataset nodes. Note that when the view only dataset nodes is turned on, the graph cannot be expanded upstream or downstream. + + * Details pane: Each lineage node has details captured and displayed when selected. + + * Dataset node has a detail pane to display all the details captured for that node for a given timestamp. Every dataset node has 3 tabs, namely: Lineage info, Schema, and History tab. The history tab lists the different versions of lineage event captured for that node. All details captured from API are displayed using metadata forms or a JSON viewer. + + * Job node has a detail pane to display job details with tabs, namely: Job info, and History. The details pane also captures query or expressions captured as part of the job run. The history tab lists the different job runs captured for that job. All details captured from API are displayed using metadata forms or a JSON viewer. + + * History tab: all lineage nodes in Amazon SageMaker Unified Studio's lineage have versioning. For every dataset node or job node, the versions are captured as history and that enables you to navigate between the different versions to identify what has changed overtime. Each version opens a new tab in the lineage page to help compare or contrast. @@ -93 +120,0 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr -## Data lineage authorization in Amazon SageMaker Unified Studio @@ -95 +121,0 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr -**Write permissions** \- to publish lineage data into Amazon SageMaker Unified Studio, you must have an IAM role with a permissions policy that includes an `ALLOW` action on the `PostLineageEvent` API. This IAM authorization happens at API Gateway layer. @@ -97 +122,0 @@ Amazon SageMaker Unified Studio’s asset details page provides a graphical repr -**Read permissions** \- there are two operations: `GetLineageNode` and `ListLineageNodeHistory` that are included in the `AmazonDataZoneDomainExecutionRolePolicy` managed policy configured by your admin. This means that every user in the Amazon SageMaker Unified Studio domain can invoke these to traverse the data lineage graph. @@ -103 +128 @@ You can use the data lineage sample experience to browse and understand data lin -Complete the followng procedure to try the sample data lineage experience in Amazon SageMaker Unified Studio: +Complete the following procedure to try the sample data lineage experience in Amazon SageMaker Unified Studio: @@ -126 +151,9 @@ At this point, a tab that provides all the space of lineage information is displ -## Automate lineage capture from Data connections +## Data lineage authorization in Amazon SageMaker Unified Studio + +**Write permissions** \- to publish lineage events into Amazon SageMaker Unified Studio, you must have an IAM role with a policy that includes an ALLOW action on the PostLineageEvent API. To publish lineage data into Amazon SageMaker Unified Studio, you must have an IAM role with a permissions policy that includes an ALLOW action on the PostLineageEvent API. This IAM authorization happens at API Gateway layer. + +**Read permissions to view lineage** \- GetLineageNode and ListLineageNodeHistory are included in the AmazonSageMakerDomainExecution managed policy and therefore every user in an Amazon SageMaker unified domain can invoke these to view the data lineage graph in Amazon SageMaker Unified Studio. + +**Read permissions to get lineage events:** you must have an IAM role with a policy that includes ALLOW action on ListLineageEvents and GetLineageEvent APIs to view lineage events posted to Amazon SageMaker Unified Studio. + +## Automate lineage capture from data connections @@ -152 +185 @@ The lineage collection in data source runs fetches information from table metada -If the number of tables is more than 100, the lineage run fails after 100 tables. Make sure that the AWS Glue crawler is not configured to bring in more that 100 tables in a run. +Lineage is captured only for crawlers which imported less than 250 tables in a crawler run. @@ -159 +192 @@ If the number of tables is more than 100, the lineage run fails after 100 tables -When enabled, the lineage runs asynchronously to capture metadata from the source and generate lineage events to be stored in SageMaker Catalog to be visualized from a particular asset. The status of lineage runs for the data source can be viewed along with data source run details. The lineage run is set up to run once daily. For the first run, after enabling the feature, the first pull is scheduled for ~5 minutes after and set for a daily run. You can configure specific time programmatically. +When enabled, the lineage runs asynchronously to capture metadata from the source and generate lineage events to be stored in SageMaker Catalog to be visualized from a particular asset. The status of lineage runs for the data source can be viewed along with data source run details. @@ -192 +225,3 @@ When enabled, the lineage runs captures queries executed for a given database an -When a new job is created in vETL in Amazon SageMaker Unified Studio, lineage is automatically enabled. When a Visual ETL flow is created, lineage capture for that ETL flow is automatically enabled when you choose **Save**. +When a new job is created in Visual ETL in Amazon SageMaker Unified Studio, lineage is automatically enabled. When a Visual ETL flow is created, lineage capture for that ETL flow is automatically enabled when you hit **Save to Project**. For every flow to capture lineage automatically, select **Save to Project** and then select **Run**. + +**Note:** if you see that lineage is not getting captured, select **Save** and then move back to the Visual ETL flows section and then reopen the Visual ETL flow. @@ -194 +229 @@ When a new job is created in vETL in Amazon SageMaker Unified Studio, lineage is -The following Spark configuration parameters are automatically added to the job being executed: +The following Spark configuration parameters are automatically added to the job being executed. When invoking Visual ETL programmatically, use the below configuration. @@ -224 +259,14 @@ The parameters are auto-configured and do not need any updates from the user. To -**Capture lineage for Spark executions in Notebooks** +**Spark compute limitations** + + * OpenLineage libraries for Spark are built into AWS Glue v5.0+ for Spark DataFrames only. Dynamic DataFrames are not supported. + + * OpenLineage libraries for Spark are built into Amazon EMR v7.5+ and only for EMR-S. + + * Capturing lineage from Spark jobs executed on EMR on EKS and EMR on EC2 are not automated but can be done by manual configuration. + + * LineageEvent has a size limit of 300KB. + + + + +**Capture lineage for AWS Glue Spark executions in Notebooks** @@ -227,0 +276 @@ Sessions in notebooks does not have a concept of a job. You can map the Spark ex +Note: for AWS Glue Spark executions in notebooks, lineage capture is automated when scheduled with workflow in shared environment using remote workflows. @@ -229 +278,2 @@ Sessions in notebooks does not have a concept of a job. You can map the Spark ex - %%configure --name project.spark -f