feat: Add async REST scan planning poll and plan storage credentials - #3724
Open
lukeFalsina wants to merge 1 commit into
Open
feat: Add async REST scan planning poll and plan storage credentials#3724lukeFalsina wants to merge 1 commit into
lukeFalsina wants to merge 1 commit into
Conversation
Catalogs that return status=submitted from planTableScan can now be polled via
GET .../plan/{plan-id}, with best-effort cancel and scan-scoped FileIO rebuilt
from plan storage-credentials. Public RestCatalog.plan_scan still returns
list[FileScanTask].
Co-authored-by: Cursor <cursoragent@cursor.com>
singhpk234
reviewed
Jul 31, 2026
singhpk234
left a comment
singhpk234
left a comment
Contributor
There was a problem hiding this comment.
Thanks @lukeFalsina this is really promising, have some suggestions inline
| [task.file.file_path for task in scan.plan_files()] | ||
| ``` | ||
|
|
||
| When the REST catalog is configured with `scan-planning-mode=server` and advertises the plan endpoint, `plan_files()` / `to_arrow()` use server-side scan planning. Catalogs that return async plans (`status=submitted`) are polled automatically; see [REST Catalog configuration](configuration.md#rest-catalog). |
Contributor
There was a problem hiding this comment.
Suggested change
| When the REST catalog is configured with `scan-planning-mode=server` and advertises the plan endpoint, `plan_files()` / `to_arrow()` use server-side scan planning. Catalogs that return async plans (`status=submitted`) are polled automatically; see [REST Catalog configuration](configuration.md#rest-catalog). | |
| When the REST catalog returns with `scan-planning-mode=server` and advertises the plan endpoint, `plan_files()` / `to_arrow()` for clients to use server-side scan planning. Catalogs that return async plans (`status=submitted`) are polled automatically till they reach terminal state; see [REST Catalog configuration](configuration.md#rest-catalog). |
| | snapshot-loading-mode | refs | The snapshots to return in the body of the metadata. Setting the value to `all` would return the full set of snapshots currently valid for the table. Setting the value to `refs` would load all snapshots referenced by branches or tags. | | ||
| | `header.X-Iceberg-Access-Delegation` | `vended-credentials` | Signal to the server that the client supports delegated access via a comma-separated list of access mechanisms. The server may choose to supply access via any or none of the requested mechanisms. When using `vended-credentials`, the server provides temporary credentials to the client. When using `remote-signing`, the server signs requests on behalf of the client. (default: `vended-credentials`) | | ||
| | view-endpoints-supported | false | For backwards compatibility with older REST servers. Set to `true` if the server supports view endpoints but doesn't send the `endpoints` field in the ConfigResponse. | | ||
| | scan-planning-mode | client | When set to `server`, and the catalog advertises the plan-table-scan endpoint, `table.scan()` uses REST server-side scan planning. Async plans (`status=submitted`) are polled via `GET .../plan/{plan-id}` until completion. | |
Contributor
There was a problem hiding this comment.
this is something loadTable return too, in case the server wants to ask client for server side scan planning ...
| storage_credentials: list[StorageCredential], | ||
| location: str | None = None, | ||
| ) -> FileIO | None: | ||
| """Build a scan-scoped FileIO from plan storage credentials. |
Contributor
There was a problem hiding this comment.
in this scenario can there be more than one plan active ?
Comment on lines
-588
to
-592
| # Collect tasks from initial response | ||
| for task in response.file_scan_tasks: | ||
| tasks.append(FileScanTask.from_rest_response(task, response.delete_files)) | ||
|
|
||
| # Fetch and collect from additional batches |
Contributor
There was a problem hiding this comment.
should we keep the comments still ?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fetchPlanningResult/cancelPlanningpolling whenplanTableScanreturnsstatus=submittedstorage-credentialsto the scan-scoped FileIO (layered on existing IO properties)RestCatalog.plan_scan(...) -> list[FileScanTask]unchanged; credentials flow through internal_plan_scan_result/_file_io_from_planRelated: #2775, #3495
Java reference: apache/iceberg#13400
Rationale
Unblocks REST catalogs that return async plans (for example policy-protected tables). Finishes the unchecked async items from #2775 and the plan-credential gap from #3495.
User-facing
table.scan()withscan-planning-mode=servernow handles async plans automaticallyrest-scan-planning.poll-timeout-ms(default 300000)RestCatalog.plan_scanreturn typeTest plan
make lintmake test(3805 passed)tests/catalog/test_scan_planning_models.pyfor poll success / timeout / failed / cancelled and IO property retentionMade with Cursor