AWS glue documentation change
Summary
Added examples and documentation for uriRegex and filterOrder tags in FileMatch rule for AWS Glue 5.0 and later, including examples for comparing files across buckets, filtering by file extension, and controlling filter order.
Security assessment
The changes are feature enhancements and documentation improvements for data quality rules in AWS Glue. They add examples and clarify usage of filtering options (uriRegex, filterOrder) but do not address any security vulnerabilities, weaknesses, or incidents. The changes are routine documentation updates for new functionality.
Diff
diff --git a/glue/latest/dg/dqdl-rule-types-FileMatch.md b/glue/latest/dg/dqdl-rule-types-FileMatch.md index ebcd33663..ed29b1fda 100644 --- a//glue/latest/dg/dqdl-rule-types-FileMatch.md +++ b//glue/latest/dg/dqdl-rule-types-FileMatch.md @@ -52,0 +53 @@ If you do not supply an algorithm, the default is SHA-256. + # Compare all files across two buckets @@ -53,0 +55 @@ If you do not supply an algorithm, the default is SHA-256. + # Compare files within specific subfolders @@ -54,0 +57,4 @@ If you do not supply an algorithm, the default is SHA-256. + # Compare only .json files across two folders + FileMatch "s3://original_bucket/" "s3://archive_bucket/" with uriRegex = "\.json$" + # Compare only the 5 most recent .csv files + FileMatch "s3://original_bucket/" "s3://archive_bucket/" with recentFiles = 5 with uriRegex = "\.csv$" with filterOrder = ["uriRegex","recentFiles"] @@ -90 +96,26 @@ This tag limits the number of files processed by keeping the most recent file fi - FileMatch "s3://amzn-s3-demo-bucket/file.json" in ["3ee0d8617ac04179sam4713e5ef8f319"] with recentFiles = 1 + FileMatch "s3://bucket/" in ["3ee0d8617ac04179sam4713e5ef8f319"] with recentFiles = 1 + +**uriRegex** + +###### Note + +The `uriRegex` tag is available in AWS Glue 5.0 and later. + +This tag filters files by applying a regex pattern to the file path. Only files whose paths match the pattern are processed. You can also use a negative lookahead to exclude files that match a pattern. + + + # Match only files with a .json extension + FileMatch "s3://bucket/" in ["3ee0d8617ac04179sam4713e5ef8f319"] with uriRegex = "\.json$" + # Exclude files ending in .tmp using a negative lookahead + FileMatch "s3://bucket/" in ["3ee0d8617ac04179sam4713e5ef8f319"] with uriRegex = "(?!.*\.tmp$).*" + +**filterOrder** + +###### Note + +The `filterOrder` tag is available in AWS Glue 5.0 and later. + +When you use multiple filter tags such as `recentFiles` and `uriRegex` together, the `filterOrder` tag controls the order in which they are applied. The default order is `recentFiles` first, then `uriRegex`. + + + FileMatch "s3://bucket/" in ["3ee0d8617ac04179sam4713e5ef8f319"] with recentFiles = 1 with uriRegex = "\.json$" with filterOrder = ["uriRegex","recentFiles"]