Document the streaming attachment transport (JS SDK) - #563
Conversation
…t SDK - Clarified the format of `localUri` for local storage references. - Added details about the `Attachment Transport` and its role in managing remote operations. - Introduced the concept of a streaming transport for large files, including configuration examples. - Updated notes on the React Native local storage adapter requirements. - Provided additional context on the `Attachment Queue` and its lifecycle management. This update aims to improve clarity and usability for developers working with attachments.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| // Optional (React Native and Node.js): a streaming transport for large files. | ||
| // It streams bytes directly between disk and network and owns | ||
| // upload/download/delete; configure it in place of remoteStorage. | ||
| // Created from the Expo, React Native FS, or Node.js local storage adapter. | ||
| // See "Transferring Large Files Without Buffering" below for a full example. | ||
| // | ||
| // const transportAdapter = localStorage.createTransportAdapter({ |
There was a problem hiding this comment.
This commented-out call is localStorage.createTransportAdapter(...), but the active localStorage in this snippet is the IndexDBFileSystemStorageAdapter declared above (web/IndexedDB). The doc states elsewhere that the web IndexedDB adapter isn't streaming-capable, so it likely doesn't expose createTransportAdapter. A reader copying this as-is would call the method on the wrong adapter. Clarify that this targets one of the React Native/Node.js adapters (currently commented out above), not the IndexedDB instance already in scope.
| class ResumableTransportAdapter implements AttachmentTransportAdapter { | ||
| async upload(attachment: LocatedAttachmentRecord): Promise<void> { | ||
| // attachment.localUri points at the source file. Transfer it to remote | ||
| // storage, e.g. in chunks that resume from the last confirmed offset | ||
| // if a previous attempt was interrupted. | ||
| } | ||
|
|
||
| async download(attachment: LocatedAttachmentRecord): Promise<void> { | ||
| // attachment.localUri is the destination path, assigned by the queue. | ||
| // Fetch the remote file into it. | ||
| } | ||
|
|
||
| async delete(attachment: AttachmentRecord): Promise<void> { | ||
| // Remove the file from remote storage. | ||
| } | ||
| } |
There was a problem hiding this comment.
All three methods here (upload, download, delete) are empty except for comments describing what to do, unlike the IPFSStorageAdapter example earlier in the file, which has real working code. Fill in at least one method with actual illustrative logic (e.g. a real fetch call or a minimal resumable-upload snippet) so this reads as a working starting point rather than pseudocode.
| The React Native local storage adapter requires Expo 54 or later. The Expo streaming transport requires Expo 56 or later. | ||
| </Warning> | ||
|
|
||
| ### Attachment Transport |
There was a problem hiding this comment.
Since attachment transport is more niche than the attachment queue, I would put this section behind the attachment queue. I think we also need to explain how transport works for other SDKs, can just be briefly, since this section currently almost makes it sound like there is no transport for the other SDKs - which doesn't make sense.
There was a problem hiding this comment.
Not is only more niche, it's also essentially a fix for a JavaScript quirk to avoid having to buffer attachments in uploads and downloads. Kotlin and Dart don't have this issue. Swift does, but doesn't have this yet.
|
|
||
| ### Attachment Transport | ||
|
|
||
| The **Attachment Transport** owns all remote operations for an attachment: upload, download, and delete. It is available in the JavaScript/TypeScript SDK only. |
There was a problem hiding this comment.
Should this be called "Transport Adapter" rather? Referring to it as "Attachment Transport" forms part of the problem I mentioned above, "what about transport in the other SDKs".
|
|
||
| Because a transport owns the entire transfer, it can stream bytes natively between the file on disk and the network without materializing them in the JS heap. PowerSync provides streaming transports for Node.js and React Native, created from their local storage adapters with `createTransportAdapter`; see [Transferring Large Files Without Buffering](#transferring-large-files-without-buffering). | ||
|
|
||
| You can configure the queue with either a `remoteStorage` or a `transportAdapter`, but not both. Supplying both, or neither, is a TypeScript compile-time error. A queue configured with a `transportAdapter` handles all remote operations through it and does not use a remote storage adapter. |
There was a problem hiding this comment.
Saying Supplying both, or neither, is a TypeScript compile-time error. feels like useless AI generated content - I think it's intuitive that there will be an error if we say "you can't do X". Or, in other words, an error is just another way of saying "you can't do X". Unless we have it here to say something about the "compile-time error" type specifically, which is not clear.
|
|
||
| By default, the queue wraps your remote storage adapter in an internal buffered transport. It reads the entire file into JS memory as an `ArrayBuffer` before handing it to the remote storage adapter, and the reverse for downloads. This works well for small files, but large files can cause memory pressure, particularly in React Native on lower-end devices. | ||
|
|
||
| To avoid this, pass an `AttachmentTransportAdapter` in the queue's `transportAdapter` option. A transport implements three methods: |
There was a problem hiding this comment.
I would refer to the example under "Configure Storage Adapters" here or move that example here.
There was a problem hiding this comment.
That example under "Configure Storage Adaptors" has different methods than mentioned here so I'm not sure whether it's referring to the same thing. The description there says "a streaming transport for large files" which I understood as what is described in this section, so we need to either consolidate or explain the difference better (and then potentially add a separate example here)
| ``` | ||
| </CodeGroup> | ||
|
|
||
| ### Transferring Large Files Without Buffering |
There was a problem hiding this comment.
Reading this section now (without understanding all the details) I almost feel like this is the best summary of the feature. I'm actually not sure whether all the above sections that refer to the transport are a duplication of this or say different things. Because it's fairly niche (that's how I understand it - just for very large files), a standalone section like this under the advanced topics actually feels the most natural and least noisy. But let me know what they thinking is behind the other sections about this above.
| The React Native local storage adapter requires Expo 54 or later. The Expo streaming transport requires Expo 56 or later. | ||
| </Warning> | ||
|
|
||
| ### Attachment Transport |
There was a problem hiding this comment.
Not is only more niche, it's also essentially a fix for a JavaScript quirk to avoid having to buffer attachments in uploads and downloads. Kotlin and Dart don't have this issue. Swift does, but doesn't have this yet.
| You can configure the queue with either a `remoteStorage` or a `transportAdapter`, but not both. Supplying both, or neither, is a TypeScript compile-time error. A queue configured with a `transportAdapter` handles all remote operations through it and does not use a remote storage adapter. | ||
|
|
||
| <Note> | ||
| The transport API requires `@powersync/web` v3.0.0, `@powersync/react-native` v2.0.3, or `@powersync/node` v0.21.0 or later. React Native also requires `@powersync/attachments-storage-react-native` v0.1.0 or later. |
There was a problem hiding this comment.
The web 3.0.0 thing is a mistake, the actual version will be 2.2.0.
|
|
||
| </CodeGroup> | ||
|
|
||
| <Note> |
There was a problem hiding this comment.
This note should be in the tab for JavaScript as it's not relevant for other SDKs.
| - **Resumable transfers** - The queue retries a failed operation by calling the transport again on the next sync interval. A transport built on a resumable protocol such as [tus](https://tus.io) or S3 multipart upload continues from the last confirmed offset instead of restarting from zero. Downloads can resume a partial file with HTTP `Range` requests | ||
| - **Encryption** - Encrypt files before upload and decrypt them after download for end-to-end encrypted attachments | ||
| - **Platform-specific transfer APIs** - Hand the transfer to an OS-level API, as the built-in React Native transports do |
There was a problem hiding this comment.
It sounds like the first two items are also possible without custom transport adapters if one implements a custom remote storage?
For the last point, maybe mention that this can be used to bypass JavaScript from downloads and uploads, instead letting a native package download directly to the file system instead? This doesn't sound like an "OS-level API" though.
3 participants
Documents the streaming attachment transport shipped in powersync-js powersync-ja/powersync-js#1039 (
@powersync/common@2.1.0), and fixes a few pre-existing issues in the JS examples found along the way.New content (all JavaScript/TypeScript only; other SDK tabs untouched):
AttachmentTransportAdapterinterface, and the either-remoteStorage-or-transportAdapter configuration rulecreateTransportAdapter, with a full Expo examplesaveFileFromUriexample for registering on-disk files without buffering, plusStreamingLocalStorageAdaptercoverage in the Local Storage Adapter sectionFixes:
state === 'SYNCED', butAttachmentStateis a numeric enum, so the checks never matchedProfilePhotoweb example passedlocal_uri(anindexeddb://reference) straight to<img src>, which fails; it now converts through the storage adapter to an object URL, with a note explaining the platform differenceThe content in this PR was AI-assisted using Claude Code.