Skip to content

[ntuple] Add RPageSourceS3 for reading ntuples from S3 - #23066

Open
JasMehta08 wants to merge 1 commit into
root-project:masterfrom
JasMehta08:s3-page-source
Open

[ntuple] Add RPageSourceS3 for reading ntuples from S3#23066
JasMehta08 wants to merge 1 commit into
root-project:masterfrom
JasMehta08:s3-page-source

Conversation

@JasMehta08

Copy link
Copy Markdown
Contributor

This Pull request:

(Is a part of the GSoC 2026 project S3 Backend for RNTuple.)
Adds RPageSourceS3.

Changes or fixes:

  • Add RPageSourceS3 implementing the basic read path: LoadStructureImpl fetches the anchor from the base URL (HEAD for the size, then GET), verifies its checksum and reads the compressed header and footer; AttachImpl decompresses and deserializes them; LoadPageListImpl and LoadClusters fetch page lists and pages.
  • Two RCurlConnection members rather than one: LoadClusters runs on the cluster pool's I/O thread while LoadSealedPageImpl runs on the caller's, and a libcurl easy handle cannot be driven by two threads at once. Splitting by role avoids a mutex, which would have held a lock across a network round trip and serialised prefetch against foreground reads.
  • AttachImpl compares the requested ntuple name against the one in the header. The name addresses nothing in S3: one base URL holds one ntuple, but since URLs are built programmatically and there is no directory listing, the comparison asserts that the URL points at the intended data set. An empty name opts out.
  • The anchor size (a server-supplied content-length) and the header and footer sizes (from the anchor) are bounded before use as allocation sizes.

Checklist:

  • tested changes locally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants