CLDSRV-898: handle checksums in CompleteMultipartUpload by leif-scality · Pull Request #6170 · scality/cloudserver

leif-scality · 2026-05-15T21:01:27Z

Calculate and compare the final object checksum with the one sent by the headers
Check that all parts have the correct checksum and checksum type
stores the final checksum when FULL_OBJECT (COMPOSITE are going to be stored by https://scality.atlassian.net/browse/S3C-10399)
lint tests/unit/api/multipartUpload.js

claude · 2026-05-15T21:05:58Z

LGTM

Review by Claude Code

codecov · 2026-05-15T21:06:39Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
9218	1	9217	0

View the full list of 2 ❄️ flaky test(s)

"before each" hook for "should retrieve a part put after part copied from MPU"::GET object With default signature With PartNumber field uploadPartCopy "before each" hook for "should retrieve a part put after part copied from MPU"

Flake rate in main: 100.00% (Passed 0 times, Failed 9 times)

Stack Traces | 6.26s run time

Connection timed out after 5000 ms

should create a bunch of objects and their versions::put and head object with versioning With v4 signature on versioning suspended then enabled bucket w/ null version should create a bunch of objects and their versions

Flake rate in main: 13.81% (Passed 668 times, Failed 107 times)

Stack Traces | 600s run time

Timeout of 600000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (.../test/versioning/objectHead.js)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

bert-e · 2026-05-21T13:19:02Z

Hello leif-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-05-21T13:19:08Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-898 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of CLDSRV-898, or the target
branch of this pull request.

claude · 2026-05-21T13:28:33Z

LGTM

Well-designed separation of CompleteMPU's final-object checksum semantics from body-digest validation. The callNext double-callback guard in processParts is a good defensive pattern for the mixed callback/promise flow. Test coverage is thorough across all algorithm/type combinations, including edge cases for default MPUs, legacy MPUs, and soft-null vs InternalError distinction.

Review by Claude Code

jonathan-gramain · 2026-05-21T18:59:02Z

+
+// XML element name AWS uses for each algorithm in CompleteMultipartUpload's
+// per-part body.
+const TAG_BY_ALGO = {


Shouldn't this be in constants too?

The algorithms object contains the TAG of each algo already, this object is just for testing that the tag was not changed in the algorithms object

Could you construct it from algorithms? If algorithm is supposed to be a constant and we are concerned that it may be accidentally changed, what about using Object.freeze on it?

The goal of the test if to raise an error if we change the xmlTag for some reason in algorithms. If we construct it from algorithms then we will never detect the change. algorithms is already Object.freeze

Then, why not using Object.freeze for each subcomponent of algorithms if the goal is to ensure that it's never modified, that would be probably a better answer than a test checking this. Just my two cents, no big deal otherwise.

claude · 2026-05-22T13:07:18Z

LGTM

The checksum handling for CompleteMultipartUpload is well-implemented: the split between md5-only body validation and final-object checksum assertion correctly matches AWS behavior. The callNext guard in processParts properly prevents double-callbacks in the mixed callback/promise flow. Per-part validation, final-object computation, and header assertion are thorough. Tests cover the full (algorithm, type) matrix including edge cases (default MPU, legacy MPU, missing parts).

Review by Claude Code

claude · 2026-05-22T13:34:18Z

completeMultipartUpload.js:352 — When storedMetadata.checksumType is undefined (legacy MPU created before the checksum feature), the error message interpolates undefined into user-facing text: "The upload was created using the undefined checksum mode." Split the !mpuType guard into a separate branch with an appropriate message.

Review by Claude Code

claude · 2026-05-22T13:37:53Z

LGTM — solid implementation with thorough test coverage.

One minor nit:
- Orphaned JSDoc in validateChecksums.js:332-337: the validateChecksumsNoChunking JSDoc now sits above the new validateContentMd5 function instead of its intended target.

Review by Claude Code

claude · 2026-05-22T14:41:36Z

require('../../../lib/services') inside an it() callback at tests/unit/api/multipartUpload.js:4119. The project testing conventions require require()/import at the top of the file, never inside describe or function bodies. Move it alongside the other top-level imports.
- Add const services = require('../../../lib/services'); at the top of the file and reference it from the test.

Review by Claude Code

claude · 2026-05-22T14:48:01Z

LGTM

The checksum validation logic is well-structured: moving CompleteMPU out of checksumedMethods into md5OnlyMethods correctly distinguishes body-digest headers from final-object-checksum headers. The callNext double-callback guard in the Promise-to-callback bridge is appropriate defensive coding. Test coverage is thorough across all algorithm/type combinations, edge cases (default vs explicit MPU, legacy MPU, missing parts), and the new x-amz-checksum-type header validation.

Review by Claude Code

claude · 2026-05-22T14:59:29Z

lib/api/completeMultipartUpload.js:623: callNext is not defined — will throw ReferenceError at runtime when computeFinalChecksum returns an error on an explicit (non-default) MPU. Should be next.
- Replace callNext(error, destBucket) with next(error, destBucket)

Review by Claude Code

claude · 2026-05-22T15:05:42Z

LGTM

Review by Claude Code

claude · 2026-05-26T15:40:52Z

LGTM

The checksum handling for CompleteMultipartUpload is well-structured: the separation of body-digest validation (md5-only path) from final-object checksum validation is clean, the per-part validation logic correctly implements the AWS semantics for default vs explicit MPUs, and the fail-or-skip strategy in computeFinalChecksum appropriately differentiates between default and explicit MPUs. Test coverage is thorough across the algorithm/type matrix, edge cases, and response validation.

Review by Claude Code

claude · 2026-05-28T10:32:07Z

LGTM

Review by Claude Code

…ing in part metadata

…igests

claude · 2026-05-28T10:49:27Z

validateContentMd5 has a dead MissingChecksum branch unreachable from its only caller (md5OnlyValidationFunc pre-checks)
- Consider making the function self-contained by removing the caller's pre-check, or documenting the precondition
computeCompositeMPUChecksum was made async but contains no await — the composite algorithms all use synchronous digest
- If intentional for consistency with computeFullObjectMPUChecksum, add a comment explaining the choice

Review by Claude Code

BourgoisMickael · 2026-06-01T12:41:43Z

-                assert(err.is.NoSuchBucket);
-                done();
-            });
+    it('should return an error on an initiate multipart upload call if ' + 'no destination bucket', done => {


Suggested change

it('should return an error on an initiate multipart upload call if ' + 'no destination bucket', done => {

it('should return an error on an initiate multipart upload call if no destination bucket', done => {

linted the whole PR, 12 changes made

BourgoisMickael · 2026-06-01T12:43:45Z

-                    partNumber: '2',
-                    uploadId: testUploadId,
+    it(
+        'should complete a multipart upload even if etag is sent ' + 'in post body without quotes (a la Cyberduck)',


Suggested change

'should complete a multipart upload even if etag is sent ' + 'in post body without quotes (a la Cyberduck)',

'should complete a multipart upload even if etag is sent in post body without quotes (a la Cyberduck)',

this happens in other places as well in this file

fixed where possible

claude · 2026-06-01T16:32:04Z

LGTM

Solid implementation of checksum handling in CompleteMultipartUpload. The separation of completeMultipartUpload from checksumedMethods into the md5OnlyMethods path is clean and correct — the x-amz-checksum-* header carries final-object semantics for CompleteMPU, not body-digest semantics. The validatePerPartChecksums, computeFinalChecksum, and validateCompleteMultipartUploadChecksum functions are well-structured with proper error paths. The async/await migration of computeCompositeMPUChecksum is necessary for the async crc64nvme.digest and is handled correctly. Test coverage is thorough across the algorithm/type matrix.

Review by Claude Code

BourgoisMickael · 2026-06-01T17:20:37Z

+                    // AWS-style message: "The {algo} you specified for part {N} did not match what we received."
+                    assert.strictEqual(
+                        err.description,
+                        `The ${wrongAlgo} you specified for part 1 did ` + 'not match what we received.',


Suggested change

`The ${wrongAlgo} you specified for part 1 did ` + 'not match what we received.',

`The ${wrongAlgo} you specified for part 1 did not match what we received.`,

I fixed other cases, the others ones where already there fixed some, some make the linter fail so they remain with '+'

claude · 2026-06-01T18:24:18Z

LGTM

Solid implementation of checksum validation and computation for CompleteMultipartUpload. The separation of CompleteMPU's x-amz-checksum-* header semantics (final-object checksum) from the body-digest validators used by other methods is clean and well-tested. The FULL_OBJECT vs COMPOSITE shape validation, the dual-form algorithm handling, and the default-vs-explicit MPU error strategy are all correct. Test coverage is comprehensive across the matrix of algorithm/type combinations, edge cases, and backward compatibility with legacy MPUs.

Review by Claude Code

claude · 2026-06-01T18:29:39Z

LGTM

Well-structured implementation with good separation of concerns: body-digest validation (md5-only path), per-part checksum validation, final-object checksum computation, and header-level final checksum assertion. Good test coverage across the AWS (algorithm, type) matrix, default MPU edge cases, and legacy backward compatibility. The defensive failOrSkip pattern (soft-null for default MPUs vs InternalError for explicit) is a clean way to handle upgrade-boundary robustness.

Review by Claude Code

leif-scality force-pushed the improvement/CLDSRV-902-calculate-final-checksum-from-parts branch 2 times, most recently from 2493909 to 538c16c Compare May 21, 2026 09:18

Base automatically changed from improvement/CLDSRV-902-calculate-final-checksum-from-parts to development/9.4 May 21, 2026 13:18

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 27b4a43 to a90bd94 Compare May 21, 2026 13:24

jonathan-gramain reviewed May 21, 2026

View reviewed changes

jonathan-gramain reviewed May 22, 2026

View reviewed changes

Comment thread lib/api/apiUtils/integrity/validateChecksums.js Outdated

Comment thread lib/api/completeMultipartUpload.js Outdated

Comment thread lib/api/completeMultipartUpload.js Outdated

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from a90bd94 to e468700 Compare May 22, 2026 13:01

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 46b6d78 to 6e735e4 Compare May 22, 2026 13:32

claude Bot reviewed May 22, 2026

View reviewed changes

Comment thread lib/api/completeMultipartUpload.js Outdated

claude Bot reviewed May 22, 2026

View reviewed changes

Comment thread lib/api/apiUtils/integrity/validateChecksums.js

CLDSRV-898: prettier format multipartUpload.js

99ee6c6

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 6e735e4 to 1c92b9b Compare May 22, 2026 14:38

claude Bot reviewed May 22, 2026

View reviewed changes

Comment thread tests/unit/api/multipartUpload.js Outdated

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 1c92b9b to 3df48c2 Compare May 22, 2026 14:44

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 3df48c2 to c190267 Compare May 22, 2026 14:55

claude Bot reviewed May 22, 2026

View reviewed changes

Comment thread lib/api/completeMultipartUpload.js Outdated

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from c190267 to 7b3785c Compare May 22, 2026 15:01

BourgoisMickael reviewed May 22, 2026

View reviewed changes

Comment thread tests/unit/api/multipartUpload.js

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from fb9139f to ee04392 Compare May 26, 2026 15:37

leif-scality requested review from BourgoisMickael and jonathan-gramain May 26, 2026 15:40

jonathan-gramain reviewed May 28, 2026

View reviewed changes

Comment thread tests/unit/api/apiUtils/integrity/validateChecksums.js Outdated

Comment thread tests/unit/api/multipartUpload.js Outdated

Comment thread tests/unit/api/multipartUpload.js Outdated

Comment thread tests/functional/aws-node-sdk/test/object/completeMpuChecksum.js Outdated

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from ee04392 to 600ef90 Compare May 28, 2026 10:27

leif-scality added 3 commits May 28, 2026 12:44

CLDSRV-898: reject Checksum<X> field on a default MPU checksum

6826d52

CLDSRV-898: reject CompleteMPU if explicit checksum but checksum miss…

2afc61e

…ing in part metadata

CLDSRV-898: stop treating x-amz-checksum-algorithm/type as checksum d…

fdb29a1

…igests

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 600ef90 to 14b8747 Compare May 28, 2026 10:46

claude Bot reviewed May 28, 2026

View reviewed changes

Comment thread lib/api/apiUtils/integrity/validateChecksums.js

claude Bot reviewed May 28, 2026

View reviewed changes

Comment thread lib/api/apiUtils/integrity/validateChecksums.js

leif-scality requested a review from jonathan-gramain May 28, 2026 10:52

BourgoisMickael reviewed Jun 1, 2026

View reviewed changes

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from 14b8747 to c39cf5f Compare June 1, 2026 16:28

leif-scality requested a review from BourgoisMickael June 1, 2026 16:50

BourgoisMickael reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/unit/api/multipartUpload.js Outdated

BourgoisMickael reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/unit/api/multipartUpload.js Outdated

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from c39cf5f to cfd50b5 Compare June 1, 2026 18:18

CLDSRV-898: rm old test config setup & remove unused string concats

a149729

leif-scality force-pushed the improvement/CLDSRV-898-complete-mpu-checksums-2 branch from cfd50b5 to a149729 Compare June 1, 2026 18:25

leif-scality requested a review from BourgoisMickael June 1, 2026 18:36

jonathan-gramain approved these changes Jun 2, 2026

View reviewed changes

	it('should return an error on an initiate multipart upload call if ' + 'no destination bucket', done => {
	it('should return an error on an initiate multipart upload call if no destination bucket', done => {

	'should complete a multipart upload even if etag is sent ' + 'in post body without quotes (a la Cyberduck)',
	'should complete a multipart upload even if etag is sent in post body without quotes (a la Cyberduck)',

	`The ${wrongAlgo} you specified for part 1 did ` + 'not match what we received.',
	`The ${wrongAlgo} you specified for part 1 did not match what we received.`,

Conversation

leif-scality commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented May 15, 2026

Uh oh!

codecov Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

bert-e commented May 21, 2026

Hello leif-scality,

Uh oh!

bert-e commented May 21, 2026

Incorrect fix version

Uh oh!

claude Bot commented May 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

claude Bot commented May 22, 2026

Uh oh!

Uh oh!

claude Bot commented May 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

claude Bot commented May 28, 2026

Uh oh!

Uh oh!

Uh oh!

claude Bot commented May 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Jun 1, 2026

Uh oh!

leif-scality commented May 15, 2026 •

edited

Loading

codecov Bot commented May 15, 2026 •

edited

Loading