Add PurgeExternalPayloads RPC for large-payload blob auto-purge#76
Draft
YunchuWang wants to merge 4 commits into
Draft
Add PurgeExternalPayloads RPC for large-payload blob auto-purge#76YunchuWang wants to merge 4 commits into
YunchuWang wants to merge 4 commits into
Conversation
Adds the authoritative bidirectional streaming RPC and its two messages (TombstonedPayload, PayloadPurged) that the backend and worker use to purge externalized large-payload blobs. The backend soft-deletes blob-externalized payload rows and streams the tombstoned tokens to a connected worker (which has storage credentials); the worker deletes each blob and acks so the backend can hard-delete the row. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pure rename of the client->server ack message; field numbers/types unchanged and wire-compatible. Reads correctly (the row is not yet purged when the worker sends it) and the ...Ack suffix signals the upstream direction in the bidi signature. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The DTFx AzureManaged SDK dials only BackendService, so it needs to drain externalized large-payload blobs via that service. Mirrors the RPC already on TaskHubSidecarService; reuses TombstonedPayload and PayloadPurgeAck, which are defined in orchestrator_service.proto (already imported here). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replaces the bidirectional streaming PurgeExternalPayloads on both TaskHubSidecarService and BackendService with two unary RPCs: GetTombstonedPayloads(limit) to fetch tombstoned large-payload rows and AckPurgedPayloads(acks) to confirm blob deletion so the backend hard-deletes those rows. Adds the four request/response messages to orchestrator_service.proto; backend_service.proto reuses them via its existing import. TombstonedPayload and PayloadPurgeAck are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
YunchuWang
added a commit
to microsoft/durabletask-dotnet
that referenced
this pull request
Jul 13, 2026
Large orchestration payloads are externalized to Azure Blob Storage as `blob:v1:<container>:<blobName>` tokens. The DTS backend stores those tokens but cannot delete the backing blobs (it has no storage credentials) — only this SDK can. This adds an opt-in, whole-scheduler singleton durable entity + orchestration job (mirroring src/ExportHistory) that drains payload rows the backend has soft-deleted and deletes their blobs, then acks so the backend can hard-delete the rows. Design: - PayloadStore.DeleteAsync is virtual (default throws NotSupportedException so it is non-breaking for existing external subclasses); BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent). - BlobPurgeJob (TaskEntity singleton): Create is a no-op when already Active so racing client processes don't disturb the running job; Run starts a fixed-id orchestrator. - BlobPurgeJobOrchestrator (perpetual): fetch a batch of tombstones, delete the blobs with capped parallelism, ack the successful deletions (failed tokens stay tombstoned to retry), idle on a timer when empty, ContinueAsNew periodically. - ExecuteBlobPurgeJobOperationOrchestrator bridges client -> entity. - Two new unary RPCs on TaskHubSidecarService: GetTombstonedPayloads / AckPurgedPayloads (authoritative proto follow-up: microsoft/durabletask-protobuf#76). - LargePayloadStorageOptions gains AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500). - Client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, without blocking host startup. Worker always registers the entity/orchestrators/activities so a client-enabled job has something to run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
YunchuWang
added a commit
to microsoft/durabletask-dotnet
that referenced
this pull request
Jul 13, 2026
Large orchestration payloads are externalized to Azure Blob Storage as `blob:v1:<container>:<blobName>` tokens. The DTS backend stores those tokens but cannot delete the backing blobs (it has no storage credentials) — only this SDK can. This adds an opt-in, whole-scheduler singleton durable entity + orchestration job (mirroring src/ExportHistory) that drains payload rows the backend has soft-deleted and deletes their blobs, then acks so the backend can hard-delete the rows. Design: - PayloadStore.DeleteAsync is virtual (default throws NotSupportedException so it is non-breaking for existing external subclasses); BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent). - BlobPurgeJob (TaskEntity singleton): Create is a no-op when already Active so racing client processes don't disturb the running job; Run starts a fixed-id orchestrator. - BlobPurgeJobOrchestrator (perpetual): fetch a batch of tombstones, delete the blobs with capped parallelism, ack the successful deletions (failed tokens stay tombstoned to retry), idle on a timer when empty, ContinueAsNew periodically. - ExecuteBlobPurgeJobOperationOrchestrator bridges client -> entity. - Two new unary RPCs on TaskHubSidecarService: GetTombstonedPayloads / AckPurgedPayloads (authoritative proto follow-up: microsoft/durabletask-protobuf#76). - LargePayloadStorageOptions gains AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500). - Client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, without blocking host startup. Worker always registers the entity/orchestrators/activities so a client-enabled job has something to run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
YunchuWang
added a commit
to microsoft/durabletask-dotnet
that referenced
this pull request
Jul 13, 2026
Large orchestration payloads are externalized to Azure Blob Storage as `blob:v1:<container>:<blobName>` tokens. The DTS backend stores those tokens but cannot delete the backing blobs (it has no storage credentials) — only this SDK can. This adds an opt-in, whole-scheduler singleton durable entity + orchestration job (mirroring src/ExportHistory) that drains payload rows the backend has soft-deleted and deletes their blobs, then acks so the backend can hard-delete the rows. Design: - PayloadStore.DeleteAsync is virtual (default throws NotSupportedException so it is non-breaking for existing external subclasses); BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent). - BlobPurgeJob (TaskEntity singleton): Create is a no-op when already Active so racing client processes don't disturb the running job; Run starts a fixed-id orchestrator. - BlobPurgeJobOrchestrator (perpetual): fetch a batch of tombstones, delete the blobs with capped parallelism, ack the successful deletions (failed tokens stay tombstoned to retry), idle on a timer when empty, ContinueAsNew periodically. - ExecuteBlobPurgeJobOperationOrchestrator bridges client -> entity. - Two new unary RPCs on TaskHubSidecarService: GetTombstonedPayloads / AckPurgedPayloads (authoritative proto follow-up: microsoft/durabletask-protobuf#76). - LargePayloadStorageOptions gains AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500). - Client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, without blocking host startup. Worker always registers the entity/orchestrators/activities so a client-enabled job has something to run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the authoritative proto definition for the large-payload blob auto-purge feature. This is the source-of-truth contract that two other PRs vendor locally and will re-sync from once this merges:
Context
When orchestration payloads exceed a threshold they are externalized to Azure Blob as tokens
blob:v1:<container>:<blobName>stored in the backend's SQLPayloads.Textcolumn. When the backend deletes that row, the blob is orphaned (the backend has no storage credentials).Fix: the backend soft-deletes blob-externalized payload rows; a connected worker (which has storage creds) receives the tombstoned tokens over a bidirectional stream, deletes each blob, and acks; the backend hard-deletes the row on ack. This RPC is that worker↔backend channel.
Change
Edits only
protos/orchestrator_service.proto:PurgeExternalPayloads(stream PayloadPurged) returns (stream TombstonedPayload)onTaskHubSidecarService, grouped with the worker-facing RPCs.TombstonedPayloadandPayloadPurgedwith exact field numbers/types matching the vendoring PRs' wire contract. Field names use camelCase to match this file's existing convention.No language stubs were regenerated: this is a proto-only repo (SDKs consume the protos via git submodule and generate code at build time into gitignored
build/dirs; no generated stubs are checked in).Draft — not ready for review until the vendoring PRs are aligned.