AWS glue documentation change
Summary
Added DataQualityGlueTable structure and updated API documentation with new fields and constraints
Security assessment
The changes introduce a new data structure and clarify API parameters for data quality operations. While CatalogId and connection details are included, there is no explicit mention of addressing security vulnerabilities or hardening access controls. The constraints on statisticId usage improve data integrity but are not security fixes.
Diff
diff --git a/glue/latest/dg/aws-glue-api-data-quality-api.md b/glue/latest/dg/aws-glue-api-data-quality-api.md index e9450fada..ca616658d 100644 --- a//glue/latest/dg/aws-glue-api-data-quality-api.md +++ b//glue/latest/dg/aws-glue-api-data-quality-api.md @@ -5 +5 @@ -— data types —DataSourceDataQualityRulesetListDetailsDataQualityTargetTableDataQualityRulesetEvaluationRunDescriptionDataQualityRulesetEvaluationRunFilterDataQualityEvaluationRunAdditionalRunOptionsDataQualityRuleRecommendationRunDescriptionDataQualityRuleRecommendationRunFilterDataQualityResultDataQualityAnalyzerResultDataQualityObservationMetricBasedObservationDataQualityMetricValuesDataQualityRuleResultDataQualityResultDescriptionDataQualityResultFilterCriteriaDataQualityRulesetFilterCriteriaDataQualityAggregatedMetricsStatisticAnnotationTimestampedInclusionAnnotationAnnotationErrorDatapointInclusionAnnotationStatisticSummaryListStatisticSummaryRunIdentifierStatisticModelResult — operations —StartDataQualityRulesetEvaluationRun (start_data_quality_ruleset_evaluation_run)CancelDataQualityRulesetEvaluationRun (cancel_data_quality_ruleset_evaluation_run)GetDataQualityRulesetEvaluationRun (get_data_quality_ruleset_evaluation_run)ListDataQualityRulesetEvaluationRuns (list_data_quality_ruleset_evaluation_runs)StartDataQualityRuleRecommendationRun (start_data_quality_rule_recommendation_run)CancelDataQualityRuleRecommendationRun (cancel_data_quality_rule_recommendation_run)GetDataQualityRuleRecommendationRun (get_data_quality_rule_recommendation_run)ListDataQualityRuleRecommendationRuns (list_data_quality_rule_recommendation_runs)GetDataQualityResult (get_data_quality_result)BatchGetDataQualityResult (batch_get_data_quality_result)ListDataQualityResults (list_data_quality_results)CreateDataQualityRuleset (create_data_quality_ruleset)DeleteDataQualityRuleset (delete_data_quality_ruleset)GetDataQualityRuleset (get_data_quality_ruleset)ListDataQualityRulesets (list_data_quality_rulesets)UpdateDataQualityRuleset (update_data_quality_ruleset)ListDataQualityStatistics (list_data_quality_statistics)TimestampFilterCreateDataQualityRulesetRequestGetDataQualityRulesetResponseGetDataQualityResultResponseStartDataQualityRuleRecommendationRunRequestGetDataQualityRuleRecommendationRunResponseBatchPutDataQualityStatisticAnnotation (batch_put_data_quality_statistic_annotation)GetDataQualityModel (get_data_quality_model)GetDataQualityModelResult (get_data_quality_model_result)ListDataQualityStatisticAnnotations (list_data_quality_statistic_annotations)PutDataQualityProfileAnnotation (put_data_quality_profile_annotation) +— data types —DataSourceDataQualityRulesetListDetailsDataQualityTargetTableDataQualityRulesetEvaluationRunDescriptionDataQualityRulesetEvaluationRunFilterDataQualityEvaluationRunAdditionalRunOptionsDataQualityRuleRecommendationRunDescriptionDataQualityRuleRecommendationRunFilterDataQualityResultDataQualityAnalyzerResultDataQualityObservationMetricBasedObservationDataQualityMetricValuesDataQualityRuleResultDataQualityResultDescriptionDataQualityResultFilterCriteriaDataQualityRulesetFilterCriteriaDataQualityAggregatedMetricsStatisticAnnotationTimestampedInclusionAnnotationAnnotationErrorDatapointInclusionAnnotationStatisticSummaryListStatisticSummaryRunIdentifierStatisticModelResultDataQualityGlueTable — operations —StartDataQualityRulesetEvaluationRun (start_data_quality_ruleset_evaluation_run)CancelDataQualityRulesetEvaluationRun (cancel_data_quality_ruleset_evaluation_run)GetDataQualityRulesetEvaluationRun (get_data_quality_ruleset_evaluation_run)ListDataQualityRulesetEvaluationRuns (list_data_quality_ruleset_evaluation_runs)StartDataQualityRuleRecommendationRun (start_data_quality_rule_recommendation_run)CancelDataQualityRuleRecommendationRun (cancel_data_quality_rule_recommendation_run)GetDataQualityRuleRecommendationRun (get_data_quality_rule_recommendation_run)ListDataQualityRuleRecommendationRuns (list_data_quality_rule_recommendation_runs)GetDataQualityResult (get_data_quality_result)BatchGetDataQualityResult (batch_get_data_quality_result)ListDataQualityResults (list_data_quality_results)CreateDataQualityRuleset (create_data_quality_ruleset)DeleteDataQualityRuleset (delete_data_quality_ruleset)GetDataQualityRuleset (get_data_quality_ruleset)ListDataQualityRulesets (list_data_quality_rulesets)UpdateDataQualityRuleset (update_data_quality_ruleset)ListDataQualityStatistics (list_data_quality_statistics)TimestampFilterCreateDataQualityRulesetRequestGetDataQualityRulesetResponseGetDataQualityResultResponseStartDataQualityRuleRecommendationRunRequestGetDataQualityRuleRecommendationRunResponseBatchPutDataQualityStatisticAnnotation (batch_put_data_quality_statistic_annotation)GetDataQualityModel (get_data_quality_model)GetDataQualityModelResult (get_data_quality_model_result)ListDataQualityStatisticAnnotations (list_data_quality_statistic_annotations)PutDataQualityProfileAnnotation (put_data_quality_profile_annotation) @@ -64,0 +65,2 @@ The Data Quality API describes the data quality data types, and includes the API + * DataQualityGlueTable structure + @@ -77,0 +80,4 @@ An AWS Glue table. + * `DataQualityGlueTable` – A DataQualityGlueTable object. + +An AWS Glue table for Data Quality Operations. + @@ -124,4 +129,0 @@ An object representing an AWS Glue table. - * `CatalogId` – UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). - -The catalog id where the AWS Glue table exists. - @@ -135,0 +138,4 @@ The name of the database where the AWS Glue table exists. + * `CatalogId` – UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +The catalog id where the AWS Glue table exists. + @@ -788,0 +795,41 @@ The inclusion annotation. +## DataQualityGlueTable structure + +The database and table in the AWS Glue Data Catalog that is used for input or output data for Data Quality Operations. + +###### Fields + + * `DatabaseName` – _Required:_ UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +A database name in the AWS Glue Data Catalog. + + * `TableName` – _Required:_ UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +A table name in the AWS Glue Data Catalog. + + * `CatalogId` – UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +A unique identifier for the AWS Glue Data Catalog. + + * `ConnectionName` – UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +The name of the connection to the AWS Glue Data Catalog. + + * `AdditionalOptions` – A map array of key-value pairs, not less than 1 or more than 10 pairs. + +Each key is a UTF-8 string, not less than 1 or more than 255 bytes long, matching the [Single-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-oneLine). + +Each value is a Description string, not more than 2048 bytes long, matching the [URI address multi-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-uri). + +Additional options for the table. Currently there are two keys supported: + + * `pushDownPredicate`: to filter on partitions without having to list and read all the files in your dataset. + + * `catalogPartitionPredicate`: to use server-side partition pruning using partition indexes in the AWS Glue Data Catalog. + + * `PreProcessingQuery` – UTF-8 string, not more than 51200 bytes long, matching the [URI address multi-line string pattern](./aws-glue-api-common.html#aws-glue-api-regex-uri). + +SQL Query of SparkSQL format that can be used to pre-process the data for the table in AWS Glue Data Catalog, before running the Data Quality Operation. + + + + @@ -2109 +2156 @@ The name of the security configuration created with the data quality encryption -Annotate datapoints over time for a specific data quality statistic. +Annotate datapoints over time for a specific data quality statistic. The API requires both profileID and statisticID as part of the InclusionAnnotation input. The API only works for a single statisticId across multiple profiles. @@ -2115 +2162 @@ Annotate datapoints over time for a specific data quality statistic. -A list of `DatapointInclusionAnnotation`'s. +A list of `DatapointInclusionAnnotation`'s. The InclusionAnnotations must contain a profileId and statisticId. If there are multiple InclusionAnnotations, the list must refer to a single statisticId across multiple profileIds.