Add CRR Cascaded capabilities by SylvainSenechal · Pull Request #6179 · scality/cloudserver

SylvainSenechal · 2026-05-29T18:00:35Z

Crr cascaded design : https://github.com/scality/citadel/pull/349

Related PRs :
Arsenal : scality/Arsenal#2628
CloudserverClient : scality/cloudserverclient#24
Backbeat : Not done yet
S3utils : scality/s3utils#395

bert-e · 2026-05-29T18:00:39Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-05-29T18:00:45Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-897 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

9.4.0

Please check the Fix Version/s of CLDSRV-897, or the target
branch of this pull request.

SylvainSenechal · 2026-05-29T18:01:03Z

+/**
+ * @param {Object} objectMD - plain object metadata (not an ObjectMD instance)
+*/
+function prepareMetadataForCascadedCrr(objectMD) {


Could be in arsenal 🤔

no i think its fine here (talking to myself)

SylvainSenechal · 2026-05-29T18:03:07Z

+                bucketName: request.bucketName,
+                objectKey: request.objectKey,
+            });
+            return callback(errors.OperationAborted);


many 409 available in arsenal. we can pick this one, maybe some other ones, or create our own

SylvainSenechal · 2026-05-29T18:04:17Z

+            // Accept both the legacy status='REPLICA' and the new isReplica flag
+            // so that cascade hops (which arrive with status='PENDING') are also
+            // recognised as replicas.
+            const isReplica = omVal.replicationInfo.isReplica === true


Should probably use the new helper function from arsenal : getReplicationIsReplica ?

claude · 2026-05-29T18:04:39Z

        "@hapi/joi": "^17.1.1",
        "@smithy/node-http-handler": "^3.0.0",
-        "arsenal": "git+https://github.com/scality/Arsenal#8.4.2",
+        "arsenal": "file:../Arsenal",


arsenal and @scality/cloudserverclient are pointing to local filesystem paths (file:../Arsenal, file:../cloudserverclient) instead of pinned git tags. This will break installs for anyone who does not have these directories locally, and CI will fail to resolve them.

Suggested change

"arsenal": "file:../Arsenal",

"arsenal": "git+https://github.com/scality/Arsenal#8.4.2",

— Claude Code

claude · 2026-05-29T18:04:46Z

    "devDependencies": {
        "@eslint/compat": "^1.2.2",
-        "@scality/cloudserverclient": "1.0.7",
+        "@scality/cloudserverclient": "file:../cloudserverclient",


Same issue: file:../cloudserverclient should be pinned to a published version or git tag.

— Claude Code

claude · 2026-05-29T18:04:55Z

+});
+
+describe('CRR cascade — putMetadata', () => {
+    it('second write with the same microVersionId returns loop-detected', async () => {


Test names using it() should start with should per project conventions.

— Claude Code

claude · 2026-05-29T18:05:03Z

+            }],
+        },
+    }));
+});


Missing after() hook to clean up the test buckets (TEST_BUCKET, TEST_BUCKET_CRR, DEST_BUCKET). Without cleanup, leftover buckets accumulate across test runs and can cause name collisions or resource leaks.

— Claude Code

bucket/keys names are randomized
I intentionnally didnt add any after to keep the test and file lighter.

Do we want to add after cleanup ?

claude · 2026-05-29T18:05:14Z

Dependency pinning: arsenal and @scality/cloudserverclient in package.json use file: paths pointing to local directories. These must be pinned to git tags or published versions before merge.
- arsenal on line 36: file:../Arsenal → should be a git tag (e.g. 8.4.2)
- @scality/cloudserverclient on line 68: file:../cloudserverclient → should be a versioned reference
Test naming: it() descriptions in tests/functional/backbeat/crrCascade.js should start with should per project conventions.
Test cleanup: No after() hook to delete the three buckets created in before(). Leftover buckets will accumulate across test runs.

Review by Claude Code

SylvainSenechal · 2026-05-29T18:05:48Z

+            if (nextReplInfo && nextReplInfo.backends.length > 0) {
+                omVal.replicationInfo = nextReplInfo;
+            } else {
+                omVal.replicationInfo = {


maybe this should be an arsenal function as its updating metadata

SylvainSenechal · 2026-05-29T18:09:19Z

all these worked locally, need to wait for the bumps now

SylvainSenechal · 2026-05-29T18:09:35Z

        "@hapi/joi": "^17.1.1",
        "@smithy/node-http-handler": "^3.0.0",
-        "arsenal": "git+https://github.com/scality/Arsenal#8.4.2",
+        "arsenal": "file:../Arsenal",


codecov · 2026-05-29T18:39:41Z

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

claude · 2026-05-29T18:40:47Z

+});
+
+describe('CRR cascade — putMetadata', () => {
+    it('second write with the same microVersionId returns loop-detected', async () => {


Test names should start with "should" per project conventions (e.g. should return loop-detected on second write with the same microVersionId).

— Claude Code

claude · 2026-05-29T18:40:55Z

        "@hapi/joi": "^17.1.1",
        "@smithy/node-http-handler": "^3.0.0",
-        "arsenal": "git+https://github.com/scality/Arsenal#8.4.2",
+        "arsenal": "file:../Arsenal",


arsenal and @scality/cloudserverclient use file: local path references. These must be pinned to a tag or version before merging — file: references will break any install outside of your local workstation.

— Claude Code

claude · 2026-05-29T18:41:03Z

package.json: arsenal and @scality/cloudserverclient use file: local path references instead of pinned tags/versions — these will break installs for anyone else and must be updated before merge
- Pin arsenal to the Arsenal PR tag once merged (was 8.4.2)
- Pin @scality/cloudserverclient to a published version (was 1.0.7)
- tests/functional/backbeat/crrCascade.js: test names in it() blocks should start with should

Review by Claude Code

claude · 2026-06-01T11:54:15Z

    }
+
+    const encodedMicroVersionId = request.headers['x-scal-micro-version-id'];
+    const incomingRaw = encodedMicroVersionId ? decodeMicroVersionId(encodedMicroVersionId) : null;


The decode error handling here differs from putMetadata (line 574-575), which normalises the result immediately so incomingRaw is guaranteed non-Error. Here incomingRaw can hold an Error object, then is checked later with !(incomingRaw instanceof Error). Consider aligning with the putMetadata pattern for consistency.

— Claude Code

claude · 2026-06-01T11:54:29Z

        "@hapi/joi": "^17.1.1",
        "@smithy/node-http-handler": "^3.0.0",
-        "arsenal": "git+https://github.com/scality/Arsenal#8.4.2",
+        "arsenal": "file:../Arsenal",


arsenal and @scality/cloudserverclient point to local file:../ paths. These must be pinned to git tags before merging — anyone cloning this repo won't have these sibling directories.

— Claude Code

claude · 2026-06-01T11:54:40Z

package.json: arsenal and @scality/cloudserverclient point to local file:../ paths — must be pinned to git tags before merging.
- lib/utilities/collectResponseHeaders.js:103: Cascade replicas without a next hop have status: '' and isReplica: true, but the outer if only checks status (falsy for empty string), so x-amz-replication-status: REPLICA header will be missing from S3 responses for these objects.
- lib/routes/routeBackbeat.js:438: putData and putMetadata handle decodeMicroVersionId error cases differently — consider aligning for consistency.
- tests/functional/backbeat/crrCascade.js: Missing after() hook to clean up the three test buckets created in before().

Review by Claude Code

Issue: CLDSRV-897

claude · 2026-06-02T22:02:05Z

+        // Old objects without microVersionId: data is already at destination.
+        // Cloudserver skips the write and returns null body with no header.
+        // Backbeat detects Location===null and treats it as partAlreadyAtDest.
+        const { versioning } = require('arsenal');


Move this require('arsenal') to the top of the file alongside the other imports. require() calls should never be inside describe or it() blocks.

— Claude Code

claude · 2026-06-02T22:02:06Z

+            }],
+        },
+    }));
+});


Missing after() hook to delete the three test buckets (TEST_BUCKET, TEST_BUCKET_CRR, DEST_BUCKET) and their objects. This leaves artifacts behind and can cause flaky runs if bucket names collide.

— Claude Code

claude · 2026-06-02T22:02:17Z

        "@aws-sdk/client-iam": "^3.930.0",
        "@aws-sdk/client-s3": "^3.1013.0",
        "@aws-sdk/client-sts": "^3.930.0",
+        "@aws-sdk/crc64-nvme-crt": "^3.989.0",


arsenal uses a file: path pointing to a sibling directory. This will not resolve in CI or for other contributors and violates dependency pinning policy (git-based deps must pin to a tag). Replace with a pinned version before merging, e.g. git+https://github.com/scality/Arsenal#8.4.2

— Claude Code

claude · 2026-06-02T22:02:18Z

-        "@aws-sdk/crc64-nvme-crt": "^3.989.0",
        "@azure/storage-blob": "^12.28.0",
        "@hapi/joi": "^17.1.1",
        "@smithy/node-http-handler": "^3.0.0",


Same issue: file: path must be replaced with a pinned version (tag or npm version) before merging.

— Claude Code

claude · 2026-06-02T22:02:28Z

package.json: arsenal and @scality/cloudserverclient use file: paths that won't resolve in CI or for other contributors. Pin to tags before merging.
- Replace file:../Arsenal with e.g. git+https://github.com/scality/Arsenal#8.4.2
- Replace file:../cloudserverclient with a pinned npm or git version
- tests/unit/routes/routeBackbeat.js:181: require('arsenal') is inside an it() callback instead of at the top of the file.
- Move it to the top-level imports
- tests/functional/backbeat/crrCascade.js: missing after() hook to clean up the three test buckets created in before().
- Add an after() that deletes objects and buckets

Review by Claude Code

SylvainSenechal commented May 29, 2026

View reviewed changes

claude Bot reviewed May 29, 2026

View reviewed changes

SylvainSenechal commented May 29, 2026

View reviewed changes

Comment thread lib/routes/routeBackbeat.js Outdated

SylvainSenechal commented May 29, 2026

View reviewed changes

SylvainSenechal mentioned this pull request May 29, 2026

Upgrade microVersionId to timestamp-ordered format and add isReplica scality/Arsenal#2628

Open

SylvainSenechal changed the title ~~Improvement/cldsrv 897~~ Add CRR Cascaded capabilities May 29, 2026

SylvainSenechal requested review from a team, delthas and maeldonn May 29, 2026 18:11

SylvainSenechal force-pushed the improvement/CLDSRV-897 branch from c15fb2b to 7912480 Compare May 29, 2026 18:38

claude Bot reviewed May 29, 2026

View reviewed changes

SylvainSenechal mentioned this pull request May 29, 2026

Added headers for CRR Cascaded scality/cloudserverclient#24

Open

SylvainSenechal force-pushed the improvement/CLDSRV-897 branch from 7912480 to 973261a Compare June 1, 2026 11:51

claude Bot reviewed Jun 1, 2026

View reviewed changes

Comment thread tests/functional/backbeat/crrCascade.js

claude Bot reviewed Jun 1, 2026

View reviewed changes

Comment thread lib/utilities/collectResponseHeaders.js Outdated

SylvainSenechal added 2 commits June 2, 2026 23:58

Implement CRR Cascaded feature

4e77798

Issue: CLDSRV-897

added tests for Crr Cascade feature

0dea4be

Issue: CLDSRV-897

SylvainSenechal force-pushed the improvement/CLDSRV-897 branch from 973261a to 0dea4be Compare June 2, 2026 21:59

claude Bot reviewed Jun 2, 2026

View reviewed changes

	"arsenal": "file:../Arsenal",
	"arsenal": "git+https://github.com/scality/Arsenal#8.4.2",

Conversation

SylvainSenechal commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented May 29, 2026

Hello sylvainsenechal,

Uh oh!

bert-e commented May 29, 2026

Incorrect fix version

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented May 29, 2026

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal May 29, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented May 29, 2026

⚠️ JUnit XML file not found

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot May 29, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented May 29, 2026

Uh oh!

claude Bot Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

claude Bot commented Jun 1, 2026

Uh oh!

claude Bot Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented May 29, 2026 •

edited

Loading

SylvainSenechal Jun 2, 2026 •

edited

Loading