AWS emr documentation change
Summary
Updated command-line option syntax from long dash (‑‑) to double hyphen (--) throughout the documentation for consistency and proper formatting.
Security assessment
The changes are purely cosmetic, replacing typographic long dashes with standard double hyphens in command-line option examples (e.g., `‑‑src` to `--src`). No security-related content was added, removed, or modified. The existing security feature documentation for `--s3ServerSideEncryption` remains unchanged in content, only the dash formatting was updated.
Diff
diff --git a/emr/latest/ReleaseGuide/UsingEMR_s3distcp.md b/emr/latest/ReleaseGuide/UsingEMR_s3distcp.md index b72baa698..8dd5d4817 100644 --- a//emr/latest/ReleaseGuide/UsingEMR_s3distcp.md +++ b//emr/latest/ReleaseGuide/UsingEMR_s3distcp.md @@ -35 +35 @@ Option | Description | Required -`‑‑src=LOCATION` | Location of the data to copy. This can be either an HDFS or Amazon S3 location. Example: `‑‑src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node` +`--src=LOCATION` | Location of the data to copy. This can be either an HDFS or Amazon S3 location. Example: `--src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node` @@ -40 +40 @@ S3DistCp does not support Amazon S3 bucket names that contain the underscore cha -`‑‑dest=LOCATION` | Destination for the data. This can be either an HDFS or Amazon S3 location. Example: `‑‑dest=hdfs:///output` +`--dest=LOCATION` | Destination for the data. This can be either an HDFS or Amazon S3 location. Example: `--dest=hdfs:///output` @@ -45,18 +45,18 @@ S3DistCp does not support Amazon S3 bucket names that contain the underscore cha -`‑‑srcPattern=PATTERN` | A [regular expression](http://en.wikipedia.org/wiki/Regular_expression) that filters the copy operation to a subset of the data at `‑‑src`. If neither `‑‑srcPattern` nor `‑‑groupBy` is specified, all data at `‑‑src` is copied to `‑‑dest`. If the regular expression argument contains special characters, such as an asterisk (*), either the regular expression or the entire `‑‑args` string must be enclosed in single quotes ('). Example: `‑‑srcPattern=.*daemons.*-hadoop-.*` | No -`‑‑groupBy=PATTERN` | A [regular expression](http://en.wikipedia.org/wiki/Regular_expression) that causes S3DistCp to concatenate files that match the expression. For example, you could use this option to combine all of the log files written in one hour into a single file. The concatenated filename is the value matched by the regular expression for the grouping. Parentheses indicate how files should be grouped, with all of the items that match the parenthetical statement being combined into a single output file. If the regular expression does not include a parenthetical statement, the cluster fails on the S3DistCp step and return an error. If the regular expression argument contains special characters, such as an asterisk (*), either the regular expression or the entire `‑‑args` string must be enclosed in single quotes ('). When `‑‑groupBy` is specified, only files that match the specified pattern are copied. You do not need to specify `‑‑groupBy` and `‑‑srcPattern` at the same time. Example: `‑‑groupBy=.*subnetid.*([0-9]+-[0-9]+-[0-9]+-[0-9]+).*` | No -`‑‑targetSize=SIZE` | The size, in mebibytes (MiB), of the files to create based on the `‑‑groupBy` option. This value must be an integer. When `‑‑targetSize` is set, S3DistCp attempts to match this size; the actual size of the copied files may be larger or smaller than this value. Jobs are aggregated based on the size of the data file, thus it is possible that the target file size will match the source data file size. If the files concatenated by `‑‑groupBy` are larger than the value of `‑‑targetSize`, they are broken up into part files, and named sequentially with a numeric value appended to the end. For example, a file concatenated into `myfile.gz` would be broken into parts as: `myfile0.gz`, `myfile1.gz`, etc. Example: `‑‑targetSize=2` | No -`‑‑appendToLastFile` | Specifies the behavior of S3DistCp when copying to files from Amazon S3 to HDFS which are already present. It appends new file data to existing files. If you use `‑‑appendToLastFile` with `‑‑groupBy`, new data is appended to files which match the same groups. This option also respects the `‑‑targetSize` behavior when used with `‑‑groupBy.` | No -`‑‑outputCodec=CODEC` | Specifies the compression codec to use for the copied files. This can take the values: `gzip`, `gz`, `lzo`, `snappy`, or `none`. You can use this option, for example, to convert input files compressed with Gzip into output files with LZO compression, or to uncompress the files as part of the copy operation. If you choose an output codec, the filename will be appended with the appropriate extension (e.g. for `gz` and `gzip`, the extension is `.gz`) If you do not specify a value for `‑‑outputCodec`, the files are copied over with no change in their compression. Example: `‑‑outputCodec=lzo` | No -`‑‑s3ServerSideEncryption` | Ensures that the target data is transferred using SSL and automatically encrypted in Amazon S3 using an AWS service-side key. When retrieving data using S3DistCp, the objects are automatically unencrypted. If you attempt to copy an unencrypted object to an encryption-required Amazon S3 bucket, the operation fails. For more information, see [Using data encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html). Example: `‑‑s3ServerSideEncryption` | No -`‑‑deleteOnSuccess` | If the copy operation is successful, this option causes S3DistCp to delete the copied files from the source location. This is useful if you are copying output files, such as log files, from one location to another as a scheduled task, and you don't want to copy the same files twice. Example: `‑‑deleteOnSuccess` | No -`‑‑disableMultipartUpload` | Disables the use of multipart upload. Example: `‑‑disableMultipartUpload` | No -`‑‑multipartUploadChunkSize=SIZE` | The size, in MiB, of each part in an Amazon S3 multipart upload. S3DistCp uses multipart upload when it copies data larger than the `multipartUploadChunkSize`. To improve job performance, you can increase the size of each part. The default size is 128 MiB. Example: `‑‑multipartUploadChunkSize=1000` | No -`‑‑numberFiles` | Prepends output files with sequential numbers. The count starts at 0 unless a different value is specified by `‑‑startingIndex`. Example: `‑‑numberFiles` | No -`‑‑startingIndex=INDEX` | Used with `‑‑numberFiles` to specify the first number in the sequence. Example: `‑‑startingIndex=1` | No -`‑‑outputManifest=FILENAME` | Creates a text file, compressed with Gzip, that contains a list of all the files copied by S3DistCp. Example: `‑‑outputManifest=manifest-1.gz` | No -`‑‑previousManifest=PATH` | Reads a manifest file that was created during a previous call to S3DistCp using the `‑‑outputManifest` flag. When the `‑‑previousManifest` flag is set, S3DistCp excludes the files listed in the manifest from the copy operation. If `‑‑outputManifest` is specified along with `‑‑previousManifest`, files listed in the previous manifest also appear in the new manifest file, although the files are not copied. Example: `‑‑previousManifest=/usr/bin/manifest-1.gz` | No -`‑‑requirePreviousManifest` | Requires a previous manifest created during a previous call to S3DistCp. If this is set to false, no error is generated when a previous manifest is not specified. The default is true. | No -`‑‑copyFromManifest` | Reverses the behavior of `‑‑previousManifest` to cause S3DistCp to use the specified manifest file as a list of files to copy, instead of a list of files to exclude from copying. Example: `‑‑copyFromManifest ‑‑previousManifest=/usr/bin/manifest-1.gz` | No -`‑‑s3Endpoint=ENDPOINT` | Specifies the Amazon S3 endpoint to use when uploading a file. This option sets the endpoint for both the source and destination. If not set, the default endpoint is `s3.amazonaws.com`. For a list of the Amazon S3 endpoints, see [Regions and endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region). Example: `‑‑s3Endpoint=s3.eu-west-1.amazonaws.com` | No -`‑‑storageClass=CLASS` | The storage class to use when the destination is Amazon S3. Valid values are STANDARD and REDUCED_REDUNDANCY. If this option is not specified, S3DistCp tries to preserve the storage class. Example: `‑‑storageClass=STANDARD` | No -`‑‑srcPrefixesFile=PATH` | a text file in Amazon S3 (s3://), HDFS (hdfs:///) or local file system (file:/) that contains a list of `src` prefixes, one prefix per line. If `srcPrefixesFile` is provided, S3DistCp will not list the src path. Instead, it generates a source list as the combined result of listing all prefixes specified in this file. The relative path as compared to src path, instead of these prefixes, will be used to generate the destination paths. If `srcPattern` is also specified, it will be applied to the combined list results of the source prefixes to further filter the input. If `copyFromManifest` is used, objects in the manifest will be copied and `srcPrefixesFile` will be ignored. Example: `‑‑srcPrefixesFile=PATH` | No +`--srcPattern=PATTERN` | A [regular expression](http://en.wikipedia.org/wiki/Regular_expression) that filters the copy operation to a subset of the data at `--src`. If neither `--srcPattern` nor `--groupBy` is specified, all data at `--src` is copied to `--dest`. If the regular expression argument contains special characters, such as an asterisk (*), either the regular expression or the entire `--args` string must be enclosed in single quotes ('). Example: `--srcPattern=.*daemons.*-hadoop-.*` | No +`--groupBy=PATTERN` | A [regular expression](http://en.wikipedia.org/wiki/Regular_expression) that causes S3DistCp to concatenate files that match the expression. For example, you could use this option to combine all of the log files written in one hour into a single file. The concatenated filename is the value matched by the regular expression for the grouping. Parentheses indicate how files should be grouped, with all of the items that match the parenthetical statement being combined into a single output file. If the regular expression does not include a parenthetical statement, the cluster fails on the S3DistCp step and return an error. If the regular expression argument contains special characters, such as an asterisk (*), either the regular expression or the entire `--args` string must be enclosed in single quotes ('). When `--groupBy` is specified, only files that match the specified pattern are copied. You do not need to specify `--groupBy` and `--srcPattern` at the same time. Example: `--groupBy=.*subnetid.*([0-9]+-[0-9]+-[0-9]+-[0-9]+).*` | No +`--targetSize=SIZE` | The size, in mebibytes (MiB), of the files to create based on the `--groupBy` option. This value must be an integer. When `--targetSize` is set, S3DistCp attempts to match this size; the actual size of the copied files may be larger or smaller than this value. Jobs are aggregated based on the size of the data file, thus it is possible that the target file size will match the source data file size. If the files concatenated by `--groupBy` are larger than the value of `--targetSize`, they are broken up into part files, and named sequentially with a numeric value appended to the end. For example, a file concatenated into `myfile.gz` would be broken into parts as: `myfile0.gz`, `myfile1.gz`, etc. Example: `--targetSize=2` | No +`--appendToLastFile` | Specifies the behavior of S3DistCp when copying to files from Amazon S3 to HDFS which are already present. It appends new file data to existing files. If you use `--appendToLastFile` with `--groupBy`, new data is appended to files which match the same groups. This option also respects the `--targetSize` behavior when used with `--groupBy.` | No +`--outputCodec=CODEC` | Specifies the compression codec to use for the copied files. This can take the values: `gzip`, `gz`, `lzo`, `snappy`, or `none`. You can use this option, for example, to convert input files compressed with Gzip into output files with LZO compression, or to uncompress the files as part of the copy operation. If you choose an output codec, the filename will be appended with the appropriate extension (e.g. for `gz` and `gzip`, the extension is `.gz`) If you do not specify a value for `--outputCodec`, the files are copied over with no change in their compression. Example: `--outputCodec=lzo` | No +`--s3ServerSideEncryption` | Ensures that the target data is transferred using SSL and automatically encrypted in Amazon S3 using an AWS service-side key. When retrieving data using S3DistCp, the objects are automatically unencrypted. If you attempt to copy an unencrypted object to an encryption-required Amazon S3 bucket, the operation fails. For more information, see [Using data encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingEncryption.html). Example: `--s3ServerSideEncryption` | No +`--deleteOnSuccess` | If the copy operation is successful, this option causes S3DistCp to delete the copied files from the source location. This is useful if you are copying output files, such as log files, from one location to another as a scheduled task, and you don't want to copy the same files twice. Example: `--deleteOnSuccess` | No +`--disableMultipartUpload` | Disables the use of multipart upload. Example: `--disableMultipartUpload` | No +`--multipartUploadChunkSize=SIZE` | The size, in MiB, of each part in an Amazon S3 multipart upload. S3DistCp uses multipart upload when it copies data larger than the `multipartUploadChunkSize`. To improve job performance, you can increase the size of each part. The default size is 128 MiB. Example: `--multipartUploadChunkSize=1000` | No +`--numberFiles` | Prepends output files with sequential numbers. The count starts at 0 unless a different value is specified by `--startingIndex`. Example: `--numberFiles` | No +`--startingIndex=INDEX` | Used with `--numberFiles` to specify the first number in the sequence. Example: `--startingIndex=1` | No +`--outputManifest=FILENAME` | Creates a text file, compressed with Gzip, that contains a list of all the files copied by S3DistCp. Example: `--outputManifest=manifest-1.gz` | No +`--previousManifest=PATH` | Reads a manifest file that was created during a previous call to S3DistCp using the `--outputManifest` flag. When the `--previousManifest` flag is set, S3DistCp excludes the files listed in the manifest from the copy operation. If `--outputManifest` is specified along with `--previousManifest`, files listed in the previous manifest also appear in the new manifest file, although the files are not copied. Example: `--previousManifest=/usr/bin/manifest-1.gz` | No +`--requirePreviousManifest` | Requires a previous manifest created during a previous call to S3DistCp. If this is set to false, no error is generated when a previous manifest is not specified. The default is true. | No +`--copyFromManifest` | Reverses the behavior of `--previousManifest` to cause S3DistCp to use the specified manifest file as a list of files to copy, instead of a list of files to exclude from copying. Example: `--copyFromManifest --previousManifest=/usr/bin/manifest-1.gz` | No +`--s3Endpoint=ENDPOINT` | Specifies the Amazon S3 endpoint to use when uploading a file. This option sets the endpoint for both the source and destination. If not set, the default endpoint is `s3.amazonaws.com`. For a list of the Amazon S3 endpoints, see [Regions and endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region). Example: `--s3Endpoint=s3.eu-west-1.amazonaws.com` | No +`--storageClass=CLASS` | The storage class to use when the destination is Amazon S3. Valid values are STANDARD and REDUCED_REDUNDANCY. If this option is not specified, S3DistCp tries to preserve the storage class. Example: `--storageClass=STANDARD` | No +`--srcPrefixesFile=PATH` | a text file in Amazon S3 (s3://), HDFS (hdfs:///) or local file system (file:/) that contains a list of `src` prefixes, one prefix per line. If `srcPrefixesFile` is provided, S3DistCp will not list the src path. Instead, it generates a source list as the combined result of listing all prefixes specified in this file. The relative path as compared to src path, instead of these prefixes, will be used to generate the destination paths. If `srcPattern` is also specified, it will be applied to the combined list results of the source prefixes to further filter the input. If `copyFromManifest` is used, objects in the manifest will be copied and `srcPrefixesFile` will be ignored. Example: `--srcPrefixesFile=PATH` | No @@ -78 +78 @@ The following example copies daemon logs from Amazon S3 to `hdfs:///output`. In - * `‑‑cluster-id` specifies the cluster + * `--cluster-id` specifies the cluster @@ -89 +89 @@ To add an S3DistCp copy step to a running cluster, put the following in a JSON f - "Args":["s3-dist-cp","‑‑s3Endpoint=s3.amazonaws.com","‑‑src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node/","‑‑dest=hdfs:///output","‑‑srcPattern=.*[a-zA-Z,]+"], + "Args":["s3-dist-cp","--s3Endpoint=s3.amazonaws.com","--src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node/","--dest=hdfs:///output","--srcPattern=.*[a-zA-Z,]+"], @@ -96 +96 @@ To add an S3DistCp copy step to a running cluster, put the following in a JSON f - aws emr add-steps ‑‑cluster-id j-3GYXXXXXX9IOK ‑‑steps file://./myStep.json + aws emr add-steps --cluster-id j-3GYXXXXXX9IOK --steps file://./myStep.json @@ -103 +103 @@ To add an S3DistCp copy step to a running cluster, put the following in a JSON f -This example also illustrates how to copy log files stored in an Amazon S3 bucket into HDFS by adding a step to a running cluster. In this example the `‑‑srcPattern` option is used to limit the data copied to the daemon logs. +This example also illustrates how to copy log files stored in an Amazon S3 bucket into HDFS by adding a step to a running cluster. In this example the `--srcPattern` option is used to limit the data copied to the daemon logs. @@ -105 +105 @@ This example also illustrates how to copy log files stored in an Amazon S3 bucke -To copy log files from Amazon S3 to HDFS using the `‑‑srcPattern` option, put the following in a JSON file saved in Amazon S3 or your local file system as ``myStep.json`` for this example. Replace `j-3GYXXXXXX9IOK` with your cluster ID and replace `amzn-s3-demo-bucket` with your Amazon S3 bucket name. +To copy log files from Amazon S3 to HDFS using the `--srcPattern` option, put the following in a JSON file saved in Amazon S3 or your local file system as ``myStep.json`` for this example. Replace `j-3GYXXXXXX9IOK` with your cluster ID and replace `amzn-s3-demo-bucket` with your Amazon S3 bucket name. @@ -111 +111 @@ To copy log files from Amazon S3 to HDFS using the `‑‑srcPattern` option, pu - "Args":["s3-dist-cp","‑‑s3Endpoint=s3.amazonaws.com","‑‑src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node/","‑‑dest=hdfs:///output","‑‑srcPattern=.*daemons.*-hadoop-.*"], + "Args":["s3-dist-cp","--s3Endpoint=s3.amazonaws.com","--src=s3://amzn-s3-demo-bucket/logs/j-3GYXXXXXX9IOJ/node/","--dest=hdfs:///output","--srcPattern=.*daemons.*-hadoop-.*"],