From f90fb37f1f3ebef6dcd1f7dadd5d6e245651399d Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 19 Jun 2026 08:18:02 +0000 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20streaming=20downloads,=20preflight,?= =?UTF-8?q?=20CLI=20peers,=20node=20config=20=E2=80=94=202026-06-19?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Track developer-facing surface changes from the ant-sdk 0.10.0, ant-client, and ant-node releases: - rest-api: implemented streaming download endpoints (POST /v1/data/stream, GET /v1/data/public/{addr}/stream) with optional NDJSON progress framing; total_chunks and already_stored_count on the prepare response - grpc-services: Stream RPC and implemented StreamPublic, DataChunk oneof and DownloadProgress, chunks_stored/payment_mode_used on the data Put responses, preflight fields on PrepareUploadResponse - mcp-server-reference: stream_download_file tool - use-external-signers: preflight fields on the prepare responses - command-reference: ant file download --peers flag - embed-a-node: drop the removed bootstrap_cache config field Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01WzRf3EPwr6UPNY36HK3rJ2 --- docs/cli/command-reference.md | 5 +- docs/mcp/mcp-server-reference.md | 12 +++-- docs/rust/embed-a-node-in-your-application.md | 5 +- ...se-external-signers-for-upload-payments.md | 18 ++++--- docs/sdk/reference/grpc-services.md | 44 ++++++++++++++-- docs/sdk/reference/rest-api.md | 51 ++++++++++++++++--- 6 files changed, 110 insertions(+), 25 deletions(-) diff --git a/docs/cli/command-reference.md b/docs/cli/command-reference.md index 9c690e3..f56f5d0 100644 --- a/docs/cli/command-reference.md +++ b/docs/cli/command-reference.md @@ -3,8 +3,8 @@ @@ -110,6 +110,7 @@ Downloads a public file by address or a private file using a local DataMap file. | `ADDRESS` | string | Conditionally | Public DataMap address. Required unless `--datamap` is provided. | | `--datamap ` | path | No | Local `.datamap` file for private download | | `-o, --output ` | path | Conditionally | Required for address-based downloads. Optional for `--datamap` downloads that can infer the original filename. | +| `--peers ` | integer | No | Number of closest peers to try for each chunk fetch. Accepts a positive integer. `--peer-count` is accepted as an alias. | **Example:** diff --git a/docs/mcp/mcp-server-reference.md b/docs/mcp/mcp-server-reference.md index c30d5f6..dd54e42 100644 --- a/docs/mcp/mcp-server-reference.md +++ b/docs/mcp/mcp-server-reference.md @@ -3,8 +3,8 @@ @@ -102,7 +102,13 @@ Uploads a local file to the network. When `private=True`, the returned `address` **Tool:** `download_file(address, dest_path, private=False)` -Downloads a file to a local path. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. +Downloads a file to a local path. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. The daemon writes the file, so the daemon and the MCP server must share a filesystem. + +### Stream a File Download + +**Tool:** `stream_download_file(address, dest_path, private=False)` + +Downloads a file to `dest_path` like `download_file`, but streams the bytes back to the MCP server process and writes them on the MCP server's host one chunk at a time with constant memory. Use it for large objects, or when the daemon and MCP server do not share a filesystem. Pass the on-network address when `private=False` (default) or the caller-held DataMap when `private=True`. The response includes `bytes_written`. ### Estimate Cost diff --git a/docs/rust/embed-a-node-in-your-application.md b/docs/rust/embed-a-node-in-your-application.md index 313fc06..8a46eb0 100644 --- a/docs/rust/embed-a-node-in-your-application.md +++ b/docs/rust/embed-a-node-in-your-application.md @@ -3,8 +3,8 @@ @@ -76,7 +76,6 @@ tokio::spawn(async move { - `upgrade` - `payment.rewards_address` - `payment.evm_network` -- `bootstrap_cache` - `storage` - `close_group_cache_dir` - `max_message_size` diff --git a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md index 442b0fa..9d147a6 100644 --- a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md +++ b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md @@ -3,15 +3,15 @@ @@ -59,11 +59,41 @@ Fetches private data using a caller-held `data_map` string. Fetches public data by address. +### Stream + +**Signature:** `Stream(StreamDataRequest) -> stream DataChunk` + +Streams private data from a caller-held `data_map` with constant memory, decrypting one batch at a time. This is the streaming counterpart of `Get` and the primitive that `StreamPublic` wraps. + +**Request fields:** + +| Name | Type | Description | +|------|------|-------------| +| `data_map` | string | Hex-encoded serialized DataMap | +| `include_progress` | bool | When `true`, the server interleaves `DownloadProgress` frames with the data frames on the same stream. Defaults to `false`, in which case the stream carries only data frames | + ### Stream Public **Signature:** `StreamPublic(StreamPublicDataRequest) -> stream DataChunk` -This RPC is exposed, but the handler returns `UNIMPLEMENTED`. +Resolves a public address to its DataMap and then streams the data, the public wrapper around `Stream`. + +**Request fields:** + +| Name | Type | Description | +|------|------|-------------| +| `address` | string | Hex data address | +| `include_progress` | bool | Same meaning as on `Stream`. Defaults to `false` | + +Each `DataChunk` frame carries exactly one of two payloads through its `kind` oneof: `data` (a decrypted plaintext batch) or `progress` (a `DownloadProgress` update). A consumer that leaves `include_progress` at `false` receives only `data` frames. + +`DownloadProgress` reports fetch progress in chunk counts: + +| Name | Type | Description | +|------|------|-------------| +| `phase` | string | One of `resolving_map`, `resolved`, or `fetching` | +| `fetched` | uint64 | Chunks fetched so far in the current phase | +| `total` | uint64 | Total chunks for the current phase, or `0` while not yet known | ### Cost @@ -183,6 +213,8 @@ Both prepare RPCs return `PrepareUploadResponse`: | `payment_vault_address` | string | Payment vault contract address (hex with `0x` prefix) | | `payment_token_address` | string | Payment token contract address (hex with `0x` prefix) | | `rpc_url` | string | EVM RPC URL for submitting transactions | +| `total_chunks` | uint64 | Full chunk count for the upload, including chunks already on-network | +| `already_stored_count` | uint64 | Chunks already stored on-network and excluded from payment and the PUT. The external signer pays for `total_chunks - already_stored_count` chunks | ### FinalizeUpload @@ -310,8 +342,10 @@ The proto files define these reusable shapes: |------|--------| | `Cost` | `atto_tokens`, `file_size`, `chunk_count`, `estimated_gas_cost_wei`, `payment_mode` | | `HealthCheckResponse` | `status`, `network`, `version`, `evm_network`, `uptime_seconds`, `build_commit`, `payment_token_address`, `payment_vault_address` | -| `PutPublicDataResponse` | `cost`, `address` | -| `PutDataResponse` | `cost`, `data_map` | +| `PutPublicDataResponse` | `cost`, `address`, `chunks_stored`, `payment_mode_used` | +| `PutDataResponse` | `cost`, `data_map`, `chunks_stored`, `payment_mode_used` | +| `DataChunk` | `kind` oneof of `data` (bytes) or `progress` (`DownloadProgress`) | +| `DownloadProgress` | `phase`, `fetched`, `total` | | `PutFileRequest` | `path`, `payment_mode` | | `PutFilePublicResponse` | `address`, `storage_cost_atto`, `gas_cost_wei`, `chunks_stored`, `payment_mode_used` | | `PutFileResponse` | `data_map`, `storage_cost_atto`, `gas_cost_wei`, `chunks_stored`, `payment_mode_used` | diff --git a/docs/sdk/reference/rest-api.md b/docs/sdk/reference/rest-api.md index f92d8e0..1c3ea75 100644 --- a/docs/sdk/reference/rest-api.md +++ b/docs/sdk/reference/rest-api.md @@ -3,8 +3,8 @@ @@ -108,7 +108,12 @@ curl http://localhost:8082/v1/data/public/ **Endpoint:** `GET /v1/data/public/{addr}/stream` -This endpoint is exposed, but the handler is a stub and returns an empty SSE stream. +Streams a public object by address with constant memory, decrypting one batch at a time instead of buffering the whole object into a JSON body. Use this for large objects. + +The response framing depends on the `Accept` header: + +- Default (any `Accept` other than `application/x-ndjson`): a raw `application/octet-stream` body of the decrypted plaintext. The `Content-Length` header is set from the object's original size, so a client detects a failed download as a short read. +- `Accept: application/x-ndjson`: newline-delimited JSON (NDJSON) frames, one JSON object per line, so the caller can drive a determinate progress bar. A leading `{"type":"meta","total_size":}` frame is followed by interleaved `{"type":"progress",...}` and `{"type":"data","chunk":""}` frames, and a terminal `{"type":"error","message":"..."}` frame if the download fails partway. Each `progress` frame carries `phase` (`"resolving_map"`, `"resolved"`, or `"fetching"`), `fetched` (chunks fetched so far), and `total` (chunks for the phase, or `0` while still unknown). **Parameters:** @@ -119,7 +124,12 @@ This endpoint is exposed, but the handler is a stub and returns an empty SSE str **Example:** ```bash -curl -N http://localhost:8082/v1/data/public//stream +# Raw bytes +curl http://localhost:8082/v1/data/public//stream -o object.bin + +# Progress framing +curl -H "Accept: application/x-ndjson" \ + http://localhost:8082/v1/data/public//stream ``` ### Store Private Data @@ -183,6 +193,29 @@ curl -X POST http://localhost:8082/v1/data/get \ -d '{"data_map":""}' ``` +### Stream Private Data + +**Endpoint:** `POST /v1/data/stream` + +Streams private data from a caller-held DataMap with constant memory, the streaming counterpart of `POST /v1/data/get`. Uses POST so the hex-encoded DataMap (which can be many KB) goes in the request body. DataMaps larger than 10 MB are rejected with `400`. + +The response framing matches `GET /v1/data/public/{addr}/stream`: a raw `application/octet-stream` body by default, or NDJSON progress frames when the request sends `Accept: application/x-ndjson`. + +**Parameters:** + +| Name | Type | Required | Description | +|------|------|----------|-------------| +| `data_map` | string | Yes | Hex-encoded serialized DataMap | + +**Example:** + +```bash +curl -X POST http://localhost:8082/v1/data/stream \ + -H "Content-Type: application/json" \ + -H "Accept: application/x-ndjson" \ + -d '{"data_map":""}' +``` + ### Estimate Data Cost **Endpoint:** `POST /v1/data/cost` @@ -613,7 +646,9 @@ The daemon returns `wave_batch` for uploads under 64 chunks and `merkle` for upl "total_amount": "", "payment_vault_address": "0x...", "payment_token_address": "0x...", - "rpc_url": "http://127.0.0.1:8545" + "rpc_url": "http://127.0.0.1:8545", + "total_chunks": 12, + "already_stored_count": 4 } ``` @@ -639,12 +674,16 @@ Merkle variant: "payment_vault_address": "0x...", "total_amount": "0", "payment_token_address": "0x...", - "rpc_url": "http://127.0.0.1:8545" + "rpc_url": "http://127.0.0.1:8545", + "total_chunks": 128, + "already_stored_count": 0 } ``` Each `pool_commitments` entry contains exactly 16 candidate payments. The example above shows one candidate for brevity. +Both variants include `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on the network; `already_stored_count` is how many of those were already stored and so excluded from payment and from the PUT. The external signer pays for `total_chunks - already_stored_count` chunks, which is why a prepared upload can cost less than the raw file size implies. + **Example:** ```bash From 140ca7a66e2c32e854407a3e835b2446a46b2995 Mon Sep 17 00:00:00 2001 From: Jim Collinson <13061030+JimCollinson@users.noreply.github.com> Date: Wed, 24 Jun 2026 16:51:21 +0100 Subject: [PATCH 2/2] docs: correct sweep source alignment --- ...se-external-signers-for-upload-payments.md | 2 +- docs/sdk/reference/grpc-services.md | 2 -- docs/sdk/reference/rest-api.md | 27 ++----------------- 3 files changed, 3 insertions(+), 28 deletions(-) diff --git a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md index 9d147a6..9f0e042 100644 --- a/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md +++ b/docs/sdk/how-to-guides/use-external-signers-for-upload-payments.md @@ -165,7 +165,7 @@ Merkle prepare response: Each `pool_commitments` entry contains exactly 16 candidate payments. The sample above shows one candidate for brevity. -Both prepare shapes also return `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on the network; `already_stored_count` is how many were already stored and so excluded from payment and the PUT. Pay for `total_chunks - already_stored_count` chunks, and reconcile against the full file size when a prepare comes back cheaper than expected. +Both prepare shapes also return `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on-network; `already_stored_count` is how many were already stored and so excluded from payment and the PUT. Pay for `total_chunks - already_stored_count` chunks, and reconcile against the full file size when a prepare comes back cheaper than expected. For file uploads, the equivalent is `POST /v1/upload/prepare` with a local `path` field instead of `data`. To make the upload publicly retrievable by address, add `"visibility":"public"` to the prepare request. `antd` bundles the serialized DataMap chunk into the same payment batch, and the finalize response includes a `data_map_address` field with its Autonomi Network address. diff --git a/docs/sdk/reference/grpc-services.md b/docs/sdk/reference/grpc-services.md index 2e8ee51..48aa99f 100644 --- a/docs/sdk/reference/grpc-services.md +++ b/docs/sdk/reference/grpc-services.md @@ -213,8 +213,6 @@ Both prepare RPCs return `PrepareUploadResponse`: | `payment_vault_address` | string | Payment vault contract address (hex with `0x` prefix) | | `payment_token_address` | string | Payment token contract address (hex with `0x` prefix) | | `rpc_url` | string | EVM RPC URL for submitting transactions | -| `total_chunks` | uint64 | Full chunk count for the upload, including chunks already on-network | -| `already_stored_count` | uint64 | Chunks already stored on-network and excluded from payment and the PUT. The external signer pays for `total_chunks - already_stored_count` chunks | ### FinalizeUpload diff --git a/docs/sdk/reference/rest-api.md b/docs/sdk/reference/rest-api.md index 1c3ea75..df827e4 100644 --- a/docs/sdk/reference/rest-api.md +++ b/docs/sdk/reference/rest-api.md @@ -193,29 +193,6 @@ curl -X POST http://localhost:8082/v1/data/get \ -d '{"data_map":""}' ``` -### Stream Private Data - -**Endpoint:** `POST /v1/data/stream` - -Streams private data from a caller-held DataMap with constant memory, the streaming counterpart of `POST /v1/data/get`. Uses POST so the hex-encoded DataMap (which can be many KB) goes in the request body. DataMaps larger than 10 MB are rejected with `400`. - -The response framing matches `GET /v1/data/public/{addr}/stream`: a raw `application/octet-stream` body by default, or NDJSON progress frames when the request sends `Accept: application/x-ndjson`. - -**Parameters:** - -| Name | Type | Required | Description | -|------|------|----------|-------------| -| `data_map` | string | Yes | Hex-encoded serialized DataMap | - -**Example:** - -```bash -curl -X POST http://localhost:8082/v1/data/stream \ - -H "Content-Type: application/json" \ - -H "Accept: application/x-ndjson" \ - -d '{"data_map":""}' -``` - ### Estimate Data Cost **Endpoint:** `POST /v1/data/cost` @@ -324,7 +301,7 @@ Prepares one raw chunk for the external-signer flow. The daemon computes the chu **Response:** -When the chunk already exists on the network: +When the chunk already exists on-network: ```json { @@ -682,7 +659,7 @@ Merkle variant: Each `pool_commitments` entry contains exactly 16 candidate payments. The example above shows one candidate for brevity. -Both variants include `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on the network; `already_stored_count` is how many of those were already stored and so excluded from payment and from the PUT. The external signer pays for `total_chunks - already_stored_count` chunks, which is why a prepared upload can cost less than the raw file size implies. +Both variants include `total_chunks` and `already_stored_count`. `total_chunks` is the full chunk count for the upload, including chunks already on-network; `already_stored_count` is how many of those were already stored and so excluded from payment and from the PUT. The external signer pays for `total_chunks - already_stored_count` chunks, which is why a prepared upload can cost less than the raw file size implies. **Example:**