AWS datazone documentation change
Summary
Modified Spark configuration parameters for OpenLineage integration, added parameter explanations, and fixed syntax formatting
Security assessment
The changes improve documentation for security-relevant configurations (e.g., specifying account IDs, domain IDs, and environment variables) but do not address a specific security vulnerability. The addition of spark.glue.accountId={ACCOUNT_ID} and explicit parameter explanations enhances security posture by clarifying secure resource identification and access control parameters.
Diff
diff --git a/datazone/latest/userguide/datazone-data-lineage.md b/datazone/latest/userguide/datazone-data-lineage.md index 3eff03da3..67be0b384 100644 --- a//datazone/latest/userguide/datazone-data-lineage.md +++ b//datazone/latest/userguide/datazone-data-lineage.md @@ -236 +236 @@ As and when AWS Glue databases and tables are added to the Amazon DataZone catal - %%configure + %%configure --name project.spark -f @@ -238,6 +238,2 @@ As and when AWS Glue databases and tables are added to the Amazon DataZone catal - “—conf”:“spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener - —conf spark.openlineage.transport.type=amazon_datazone_api - —conf spark.openlineage.transport.domainId=<datazone domainID> - —conf spark.openlineage.facets.custom_environment_variables [AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;] - —conf spark.glue.JobName=<SessionId> - —conf spark.glue.JobRunId=<SessionId or NONE?>“ (as session is a resource and doesn't have subsequent runs - interactive) + "--conf":"spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener --conf spark.openlineage.transport.type=amazon_datazone_api --conf spark.openlineage.transport.domainId={DOMAIN_ID} --conf spark.glue.accountId={ACCOUNT_ID} --conf spark.openlineage.facets.custom_environment_variables=[AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION; --conf spark.glue.JOB_NAME={JOB_NAME}" + } @@ -246 +242,13 @@ As and when AWS Glue databases and tables are added to the Amazon DataZone catal -Note: there are 2 dashes in front of conf - quip is updating to hyphen. +The following are the parameter details: + + * `spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener` \- OpenLineageSparkListener will be created and registered with Spark's listener bus + + * `spark.openlineage.transport.type=amazon_datazone_api` \- This is an OpenLineage specification to tell the OpenLineage Plugin to use DataZone API Transport to emit lineage events to DataZone’s PostLineageEvent API. For more information, see [https://openlineage.io/docs/integrations/spark/configuration/spark_conf](https://openlineage.io/docs/integrations/spark/configuration/spark_conf) + + * `spark.openlineage.transport.domainId={DOMAIN_ID}` \- This parameter establishes the domain to which the API transport will submit the lineage events to. + + * `spark.openlineage.facets.custom_environment_variables [AWS_DEFAULT_REGION;GLUE_VERSION;GLUE_COMMAND_CRITERIA;GLUE_PYTHON_VERSION;]` \- The following environment variables (AWS_DEFAULT_REGION , GLUE_VERSION , GLUE_COMMAND_CRITERIA, and GLUE_PYTHON_VERSION), which Glue interactive session populates, will be added to the LineageEvent + + * `spark.glue.accountId=<ACCOUNT_ID>` \- Account Id of the Glue Data Catalog where the metadata resides. This account id is used to construct Glue ARN in lineage event. + + * `spark.glue.JOB_NAME` \- Job name of the lineage event. The job name in notebook can be set as `spark.glue.JOB_NAME: ${projectId}.${pathToNotebook}`.