AWS AmazonS3 documentation change
Summary
Significant restructuring of checksum documentation including: removed detailed technical explanations of multipart upload validation, simplified algorithm listings, removed code examples, and reorganized content into separate topics. Added emphasis on CRC64NVME as default checksum algorithm and MD5 support.
Security assessment
The changes primarily restructure existing checksum documentation rather than address a specific vulnerability. However, they add explicit documentation about using checksums (including MD5) for data integrity validation, which is a security feature. No evidence of addressing a specific security vulnerability is present in the diff.
Diff
diff --git a/AmazonS3/latest/userguide/checking-object-integrity.md b/AmazonS3/latest/userguide/checking-object-integrity.md index b6495b5ca..60c0f0ea6 100644 --- a//AmazonS3/latest/userguide/checking-object-integrity.md +++ b//AmazonS3/latest/userguide/checking-object-integrity.md @@ -5,2 +4,0 @@ -Using supported checksum algorithmsFull object and composite checksum typesUsing full object checksums for multipart uploadUsing part-level checksums for multipart uploadChecksum operationsUsing Content-MD5 when uploading objectsUsing Content-MD5 and the ETag to verify uploaded objectsUsing trailing checksums - @@ -9,3 +7 @@ Using supported checksum algorithmsFull object and composite checksum typesUsing -Amazon S3 uses checksum values to verify the integrity of data that you upload or download. In addition, you can request that another checksum value be calculated for any object that you store in Amazon S3. You can choose a checksum algorithm to use when uploading, copying, or batch copying your data. - -When you upload your data, Amazon S3 uses the algorithm that you've chosen to compute a checksum on the server side and validates it with the provided value before storing the object and storing the checksum as part of the object metadata. This validation works consistently across encryption modes, object sizes, and storage classes for both single part and multipart uploads. When you copy or batch copy your data, however, Amazon S3 calculates the checksum on the source object and moves it to the destination object. +Amazon S3 provides a range of data protection capabilities throughout an object's storage lifecycle. With Amazon S3, you can use checksum values to verify the integrity of the data that you upload or download. In addition, you can request that another checksum value be calculated for any object that you store in S3. @@ -13 +9 @@ When you upload your data, Amazon S3 uses the algorithm that you've chosen to co -###### Note +When uploading, copying, or managing your data, you can choose from several supported checksum algorithms: @@ -15,3 +11 @@ When you upload your data, Amazon S3 uses the algorithm that you've chosen to co -When you perform a single part or multipart upload, you can optionally include a precalculated checksum as part of your request, and use the full object checksum type. To use precalculated values with multiple objects, use the AWS CLI or AWS SDKs. - -## Using supported checksum algorithms + * CRC-64/NVME (`CRC64NVME`) @@ -19 +13 @@ When you perform a single part or multipart upload, you can optionally include a -With Amazon S3, you can choose a checksum algorithm to validate your data during uploads. The specified checksum algorithm is then stored with your object and can be used to validate data integrity during downloads. You can choose one of the following Secure Hash Algorithms (SHA) or Cyclic Redundancy Check (CRC) checksum algorithms to calculate the checksum value: +###### Note @@ -21 +15 @@ With Amazon S3, you can choose a checksum algorithm to validate your data during - * CRC-64/NVME (`CRC64NVME`) +The `CRC64NVME` checksum algorithm is the default checksum algorithm used for checksum calculations. @@ -29,22 +23 @@ With Amazon S3, you can choose a checksum algorithm to validate your data during - * SHA-256 (`SHA256`) (SHA-2 family) - - - - -Additionally, you can provide a checksum with each request using the Content-MD5 header. - -When you [upload an object](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html), you specify the algorithm that you want to use: - - * **When you use the AWS Management Console** , choose the checksum algorithm that you want to use. You can optionally specify the checksum value of the object. When Amazon S3 receives the object, it calculates the checksum by using the algorithm that you specified. If the two checksum values don't match, Amazon S3 generates an error. - - * **When you use an SDK** , be aware of the following: - - * Set the `ChecksumAlgorithm` parameter to the algorithm that you want Amazon S3 to use. If you already have a precalculated checksum, you pass the checksum value to the AWS SDK, and the SDK includes the value in the request. If you don’t pass a checksum value or don’t specify a checksum algorithm, the SDK automatically calculates a checksum value for you and includes it with the request to provide integrity protections. If the individual checksum value doesn't match the set value of the checksum algorithm, Amazon S3 fails the request with a `BadDigest` error. - - * If you’re using an upgraded AWS SDK, the SDK chooses a checksum algorithm for you. However, you can override this checksum algorithm. - - * If you don’t specify a checksum algorithm and the SDK also doesn’t calculate a checksum for you, then S3 automatically chooses the CRC-64/NVME (`CRC64NVME`) checksum algorithm. - - * **When you use the REST API** , don't use the `x-amz-sdk-checksum-algorithm` parameter. Instead, use one of the algorithm-specific headers (for example, `x-amz-checksum-crc32`). - - + * SHA-256 (`SHA256`) @@ -52,18 +25 @@ When you [upload an object](https://docs.aws.amazon.com/AmazonS3/latest/userguid - -To apply any of these checksum values to objects that are already uploaded to Amazon S3, you can copy the object and specify whether you want to use the existing checksum algorithm or a new one. If you don’t specify an algorithm, S3 uses the existing algorithm. If the source object doesn’t have a specified checksum algorithm or checksum value, Amazon S3 uses the CRC-64/NVME algorithm to calculate the checksum value for the destination object. You can also specify a checksum algorithm when copying objects using [S3 Batch Operations](https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops.html). - -###### Important - -If you use a multipart upload with **Checksums** for composite (or part-level) checksums, the multipart upload part numbers must be consecutive and begin with 1. If you try to complete a multipart upload request with nonconsecutive part numbers, Amazon S3 generates an `HTTP 500 Internal Server` error. - -## Full object and composite checksum types - -In Amazon S3, there are two types of supported checksums: - - * **Full object checksums:** A full object checksum is calculated based on all of the content of a multipart upload, covering all data from the first byte of the first part to the last byte of the last part. - -###### Note - -All `PUT` requests require a full object checksum type. - - * **Composite checksums:** A composite checksum is calculated based on the individual checksums of each part in a multipart upload. Instead of computing a checksum based on all of the data content, this approach aggregates the part-level checksums (from the first part to the last) to produce a single, combined checksum for the complete object. + * MD5 (`MD5`) @@ -73,370 +29 @@ All `PUT` requests require a full object checksum type. -When an object is uploaded as a multipart upload, the entity tag (ETag) for the object is not an MD5 digest of the entire object. Instead, Amazon S3 calculates the MD5 digest of each individual part as it is uploaded. The MD5 digests are used to determine the ETag for the final object. Amazon S3 concatenates the bytes for the MD5 digests together and then calculates the MD5 digest of these concatenated values. During the final ETag creation step, Amazon S3 adds a dash with the total number of parts to the end. - - - - -Amazon S3 supports the following full object and composite checksum algorithm types: - - * CRC-64/NVME (`CRC64NVME`): Supports the full object algorithm type only. - - * CRC-32 (`CRC32`): Supports both full object and composite algorithm types. - - * CRC-32C (`CRC32C`): Supports both full object and composite algorithm types. - - * SHA-1 (`SHA1`): Supports both full object and composite algorithm types. - - * SHA-256 (`SHA256`): Supports both full object and composite algorithm types. - - - - -###### Note - -If you're using a `SHA-1` or `SHA-256` checksum algorithm for a multipart upload, you must use the composite algorithm type. If you're using a `SHA-1` or `SHA-256` checksum algorithm for a single part upload, only the full object algorithm type is supported. - -### Single part uploads - -Checksums of objects that are uploaded in a single part (using [PutObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject)) are treated as full object checksums. When you upload an object in the Amazon S3 console, you can choose the checksum algorithm that you want S3 to use and also (optionally) provide a precomputed value. Amazon S3 then validates this checksum before storing the object and its checksum value. You can verify an object's data integrity when you request the checksum value during object downloads. - -### Multipart uploads - -When you upload the object in multiple parts using the [MultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html) API, you can specify the checksum algorithm that you want Amazon S3 to use and the checksum type (full object or composite). - -The following table indicates which checksum algorithm type is supported for each checksum algorithm in a multipart upload: - -Checksum algorithm | Full object | Composite ----|---|--- -CRC-64/NVME (`CRC64NVME`) | Yes | No -CRC-32 (`CRC32`) | Yes | Yes -CRC-32C (`CRC32C`) | Yes | Yes -SHA-1 (`SHA1`) | No | Yes -SHA-256 (`SHA256`) | No | Yes - -## Using full object checksums for multipart upload - -When creating or performing a multipart upload, you can use full object checksums for validation on upload. This means that you can provide the checksum algorithm for the [MultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_MultipartUpload.html) API, simplifying your integrity validation tooling because you no longer need to track part boundaries for uploaded objects. You can provide the checksum of the whole object in the [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) request, along with the object size. - -When you provide a full object checksum during a multipart upload, the AWS SDK passes the checksum to Amazon S3, and S3 validates the object integrity server-side, comparing it to the received value. Then, Amazon S3 stores the object if the values match. If the two values don’t match, S3 fails the request with a `BadDigest` error. The checksum of your object is also stored in object metadata that you use later to validate an object's data integrity. - -For full object checksums, you can use CRC-64/NVME (`CRC64NVME`), CRC-32 (`CRC32`), or CRC-32C (`CRC32C`) checksum algorithms in S3. Full object checksums in multipart uploads are only available for CRC-based checksums because they can linearize into a full object checksum. This linearization allows Amazon S3 to parallelize your requests for improved performance. In particular, S3 can compute the checksum of the whole object from the part-level checksums. This type of validation isn’t available for other algorithms, such as SHA and MD5. Because S3 has default integrity protections, if objects are uploaded without a checksum, S3 automatically attaches the recommended full object CRC-64/NVME (`CRC64NVME`) checksum algorithm to the object. - -###### Note - -To initiate the multipart upload, you can specify the checksum algorithm and the full object checksum type. After you specify the checksum algorithm and the full object checksum type, you can provide the full object checksum value for the multipart upload. - -## Using part-level checksums for multipart upload - -When objects are uploaded to Amazon S3, they can be uploaded as a single object or uploaded in parts with the multipart upload process. You can choose a **Checksum** type for your multipart upload. For multipart upload part-level checksums (or composite checksums), Amazon S3 calculates the checksum for each individual part by using the specified checksum algorithm. You can use [`UploadPart`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) to provide the checksum values for each part. If the object that you try to upload in the Amazon S3 console is set to use the CRC-64/NVME (`CRC64NVME`) checksum algorithm and exceeds 16 MB, it is automatically designated as a full object checksum. - -Amazon S3 then uses the stored part-level checksum values to confirm that each part is uploaded correctly. When each part’s checksum (for the whole object) is provided, S3 uses the stored checksum values of each part to calculate the full object checksum internally, comparing it with the provided checksum value. This minimizes compute costs since S3 can compute a checksum of the whole object using the checksum of the parts. For more information about multipart uploads, see [Uploading and copying objects using multipart upload in Amazon S3](./mpuoverview.html) and Using full object checksums for multipart upload. - -When the object is completely uploaded, you can use the final calculated checksum to verify the data integrity of the object. - -When uploading a part of the multipart upload, be aware of the following: - - * To retrieve information about the object, including how many parts make up the entire object, you can use the [GetObjectAttributes](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html) operation. With additional checksums, you can also recover information for each individual part that includes the part's checksum value. - - * For completed uploads, you can get an individual part's checksum by using the [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) or [HeadObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html) operations and specifying a part number or byte range that aligns with a single part. If you want to retrieve the checksum values for individual parts of multipart uploads that are still in progress, you can use [ListParts](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html). - - * Because of how Amazon S3 calculates the checksum for multipart objects, the checksum value for the object might change if you copy it. If you're using an SDK or the REST API and you call [CopyObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html), Amazon S3 copies any object up to the size limitations of the `CopyObject` API operation. Amazon S3 does this copy as a single action, regardless of whether the object was uploaded in a single request or as part of a multipart upload. With a copy command, the checksum of the object is a direct checksum of the full object. If the object was originally uploaded using a multipart upload, the checksum value changes even though the data doesn't. - - * Objects that are larger than the size limitations of the `CopyObject` API operation must use [multipart upload copy commands](https://docs.aws.amazon.com/AmazonS3/latest/userguide/CopyingObjectsMPUapi.html). - - * When you perform some operations using the AWS Management Console, Amazon S3 uses a multipart upload if the object is greater than 16 MB in size. - - - - -## Checksum operations - -After uploading objects, you can get the checksum value and compare it to a precomputed or previously stored checksum value of the same algorithm type. The following examples show you which checksum operations or methods you can use to verify data integrity. - -To learn more about using the console and specifying checksum algorithms to use when uploading objects, see [Uploading objects](./upload-objects.html) and [Tutorial: Checking the integrity of data in Amazon S3 with additional checksums](https://aws.amazon.com/getting-started/hands-on/amazon-s3-with-additional-checksums/?ref=docs_gateway/amazons3/checking-object-integrity.html). - -The following example shows how you can use the AWS SDKs to upload a large file with multipart upload, download a large file, and validate a multipart upload file, all by using SHA-256 for file validation. - -Java - - -###### Example: Uploading, downloading, and verifying a large file with SHA-256 - -For instructions on creating and testing a working sample, see [Getting Started](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/getting-started.html) in the AWS SDK for Java Developer Guide. - - - import software.amazon.awssdk.auth.credentials.AwsCredentials; - import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; - import software.amazon.awssdk.core.ResponseInputStream; - import software.amazon.awssdk.core.sync.RequestBody; - import software.amazon.awssdk.regions.Region; - import software.amazon.awssdk.services.s3.S3Client; - import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest; - import software.amazon.awssdk.services.s3.model.ChecksumAlgorithm; - import software.amazon.awssdk.services.s3.model.ChecksumMode; - import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest; - import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse; - import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload; - import software.amazon.awssdk.services.s3.model.CompletedPart; - import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest; - import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; - import software.amazon.awssdk.services.s3.model.GetObjectAttributesRequest; - import software.amazon.awssdk.services.s3.model.GetObjectAttributesResponse; - import software.amazon.awssdk.services.s3.model.GetObjectRequest; - import software.amazon.awssdk.services.s3.model.GetObjectResponse; - import software.amazon.awssdk.services.s3.model.GetObjectTaggingRequest; - import software.amazon.awssdk.services.s3.model.ObjectAttributes; - import software.amazon.awssdk.services.s3.model.PutObjectTaggingRequest; - import software.amazon.awssdk.services.s3.model.Tag; - import software.amazon.awssdk.services.s3.model.Tagging; - import software.amazon.awssdk.services.s3.model.UploadPartRequest; - import software.amazon.awssdk.services.s3.model.UploadPartResponse; - - import java.io.File; - import java.io.FileInputStream; - import java.io.FileOutputStream; - import java.io.IOException; - import java.io.InputStream; - import java.io.OutputStream; - import java.nio.ByteBuffer; - import java.security.MessageDigest; - import java.security.NoSuchAlgorithmException;