diff --git a/docs/cli/command-reference.md b/docs/cli/command-reference.md index 9c690e3..f56f5d0 100644 --- a/docs/cli/command-reference.md +++ b/docs/cli/command-reference.md @@ -3,8 +3,8 @@ @@ -110,6 +110,7 @@ Downloads a public file by address or a private file using a local DataMap file. | `ADDRESS` | string | Conditionally | Public DataMap address. Required unless `--datamap` is provided. | | `--datamap ` | path | No | Local `.datamap` file for private download | | `-o, --output ` | path | Conditionally | Required for address-based downloads. Optional for `--datamap` downloads that can infer the original filename. | +| `--peers ` | integer | No | Number of closest peers to try for each chunk fetch. Accepts a positive integer. `--peer-count` is accepted as an alias. | **Example:** diff --git a/docs/mcp/mcp-server-reference.md b/docs/mcp/mcp-server-reference.md index c30d5f6..dd54e42 100644 --- a/docs/mcp/mcp-server-reference.md +++ b/docs/mcp/mcp-server-reference.md @@ -3,8 +3,8 @@ @@ -102,7 +102,13 @@ Uploads a local file to the network. When `private=True`, the returned `address` **Tool:** `download_file(address, dest_path, private=False)` -Downloads a file to a local path. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. +Downloads a file to a local path. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. The daemon writes the file, so the daemon and the MCP server must share a filesystem. + +### Stream a File Download + +**Tool:** `stream_download_file(address, dest_path, private=False)` + +Downloads a file to `dest_path` like `download_file`, but streams the bytes back to the MCP server process and writes them on the MCP server's host one chunk at a time with constant memory. Use it for large objects, or when the daemon and MCP server do not share a filesystem. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. The response includes `bytes_written`. ### Estimate Cost diff --git a/docs/rust/embed-a-node-in-your-application.md b/docs/rust/embed-a-node-in-your-application.md index 313fc06..8a46eb0 100644 --- a/docs/rust/embed-a-node-in-your-application.md +++ b/docs/rust/embed-a-node-in-your-application.md @@ -3,8 +3,8 @@ @@ -76,7 +76,6 @@ tokio::spawn(async move { - `upgrade` - `payment.rewards_address` - `payment.evm_network` -- `bootstrap_cache` - `storage` - `close_group_cache_dir` - `max_message_size` diff --git a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md index 442b0fa..9f0e042 100644 --- a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md +++ b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md @@ -3,15 +3,15 @@ @@ -59,11 +59,41 @@ Fetches private data using a caller-held `data_map` string. Fetches public data by address. +### Stream + +**Signature:** `Stream(StreamDataRequest) -> stream DataChunk` + +Streams private data from a caller-held `data_map` with constant memory, decrypting one batch at a time. This is the streaming counterpart of `Get` and the primitive that `StreamPublic` wraps. + +**Request fields:** + +| Name | Type | Description | +|------|------|-------------| +| `data_map` | string | Hex-encoded serialized DataMap | +| `include_progress` | bool | When `true`, the server interleaves `DownloadProgress` frames with the data frames on the same stream. Defaults to `false`, in which case the stream carries only data frames | + ### Stream Public **Signature:** `StreamPublic(StreamPublicDataRequest) -> stream DataChunk` -This RPC is exposed, but the handler returns `UNIMPLEMENTED`. +Resolves a public address to its DataMap and then streams the data, the public wrapper around `Stream`. + +**Request fields:** + +| Name | Type | Description | +|------|------|-------------| +| `address` | string | Hex data address | +| `include_progress` | bool | Same meaning as on `Stream`. Defaults to `false` | + +Each `DataChunk` frame carries exactly one of two payloads through its `kind` oneof: `data` (a decrypted plaintext batch) or `progress` (a `DownloadProgress` update). A consumer that leaves `include_progress` at `false` receives only `data` frames. + +`DownloadProgress` reports fetch progress in chunk counts: + +| Name | Type | Description | +|------|------|-------------| +| `phase` | string | One of `resolving_map`, `resolved`, or `fetching` | +| `fetched` | uint64 | Chunks fetched so far in the current phase | +| `total` | uint64 | Total chunks for the current phase, or `0` while not yet known | ### Cost @@ -310,8 +340,10 @@ The proto files define these reusable shapes: |------|--------| | `Cost` | `atto_tokens`, `file_size`, `chunk_count`, `estimated_gas_cost_wei`, `payment_mode` | | `HealthCheckResponse` | `status`, `network`, `version`, `evm_network`, `uptime_seconds`, `build_commit`, `payment_token_address`, `payment_vault_address` | -| `PutPublicDataResponse` | `cost`, `address` | -| `PutDataResponse` | `cost`, `data_map` | +| `PutPublicDataResponse` | `cost`, `address`, `chunks_stored`, `payment_mode_used` | +| `PutDataResponse` | `cost`, `data_map`, `chunks_stored`, `payment_mode_used` | +| `DataChunk` | `kind` oneof of `data` (bytes) or `progress` (`DownloadProgress`) | +| `DownloadProgress` | `phase`, `fetched`, `total` | | `PutFileRequest` | `path`, `payment_mode` | | `PutFilePublicResponse` | `address`, `storage_cost_atto`, `gas_cost_wei`, `chunks_stored`, `payment_mode_used` | | `PutFileResponse` | `data_map`, `storage_cost_atto`, `gas_cost_wei`, `chunks_stored`, `payment_mode_used` | diff --git a/docs/sdk/reference/rest-api.md b/docs/sdk/reference/rest-api.md index f92d8e0..df827e4 100644 --- a/docs/sdk/reference/rest-api.md +++ b/docs/sdk/reference/rest-api.md @@ -3,8 +3,8 @@ @@ -108,7 +108,12 @@ curl http://localhost:8082/v1/data/public/ **Endpoint:** `GET /v1/data/public/{addr}/stream` -This endpoint is exposed, but the handler is a stub and returns an empty SSE stream. +Streams a public object by address with constant memory, decrypting one batch at a time instead of buffering the whole object into a JSON body. Use this for large objects. + +The response framing depends on the `Accept` header: + +- Default (any `Accept` other than `application/x-ndjson`): a raw `application/octet-stream` body of the decrypted plaintext. The `Content-Length` header is set from the object's original size, so a client detects a failed download as a short read. +- `Accept: application/x-ndjson`: newline-delimited JSON (NDJSON) frames, one JSON object per line, so the caller can drive a determinate progress bar. A leading `{"type":"meta","total_size":}` frame is followed by interleaved `{"type":"progress",...}` and `{"type":"data","chunk":""}` frames, and a terminal `{"type":"error","message":"..."}` frame if the download fails partway. Each `progress` frame carries `phase` (`"resolving_map"`, `"resolved"`, or `"fetching"`), `fetched` (chunks fetched so far), and `total` (chunks for the phase, or `0` while still unknown). **Parameters:** @@ -119,7 +124,12 @@ This endpoint is exposed, but the handler is a stub and returns an empty SSE str **Example:** ```bash -curl -N http://localhost:8082/v1/data/public//stream +# Raw bytes +curl http://localhost:8082/v1/data/public//stream -o object.bin + +# Progress framing +curl -H "Accept: application/x-ndjson" \ + http://localhost:8082/v1/data/public//stream ``` ### Store Private Data @@ -291,7 +301,7 @@ Prepares one raw chunk for the external-signer flow. The daemon computes the chu **Response:** -When the chunk already exists on the network: +When the chunk already exists on-network: ```json { @@ -613,7 +623,9 @@ The daemon returns `wave_batch` for uploads under 64 chunks and `merkle` for upl "total_amount": "", "payment_vault_address": "0x...", "payment_token_address": "0x...", - "rpc_url": "http://127.0.0.1:8545" + "rpc_url": "http://127.0.0.1:8545", + "total_chunks": 12, + "already_stored_count": 4 } ``` @@ -639,12 +651,16 @@ Merkle variant: "payment_vault_address": "0x...", "total_amount": "0", "payment_token_address": "0x...", - "rpc_url": "http://127.0.0.1:8545" + "rpc_url": "http://127.0.0.1:8545", + "total_chunks": 128, + "already_stored_count": 0 } ``` Each `pool_commitments` entry contains exactly 16 candidate payments. The example above shows one candidate for brevity. +Both variants include `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on-network; `already_stored_count` is how many of those were already stored and so excluded from payment and from the PUT. The external signer pays for `total_chunks - already_stored_count` chunks, which is why a prepared upload can cost less than the raw file size implies. + **Example:** ```bash