runtime: core reliability — definitive resume, teardown + task-retention fixes (PR #122 stage D)#136
Merged
Merged
Conversation
…ion fixes (PR #122 stage D) Bucket-A slice of the origin/abridge/gateway-tito reliability work (commit b81f2fc), ported surgically onto master with NO wire-contract removals (the HTTP /call fast path, the /log structured bridge, and the codec ext types all stay; those belong to later stages of the #122 split). Server (runtime/server/sio.py): - resume is now definitive: a cached result replays, a running call is left alone, and an evicted/unknown call_id gets call:error type=ResultUnavailable — never silence that hangs the host's remote() forever. - the completed-task transition is atomic w.r.t. resume: the result is stored into pending_results synchronously inside the done-callback (same loop turn as the calls.pop), so a resume can never observe a completing call_id in neither map and misreport a live result as unavailable. (This window existed on the source branch; ported as-fixed.) - result-emit tasks are strongly referenced (emit_tasks set) — a bare create_task in a done-callback can be GC'd before it runs, silently dropping a completed call's result. Applied to both the SIO path and the retained HTTP fast path's timeout branch. Client (runtime/client/client.py): - _ensure_sio tears down a disconnected stale handle before building a fresh one — overwriting it leaked the aiohttp session + reconnect task and could leave a second live /rpc socket. - a malformed call:error with no error payload resolves to a typed MalformedError instead of a bare KeyError escaping remote(). - server-side cancellation now raises CallCancelled (RemoteCallError subclass), not a bare asyncio.CancelledError that reads as local task cancellation. - new try_remote() -> Result[R] (Ok | Failed) for harnesses that branch on outcomes at scale; also on Sandbox. Provider (provider/base.py): - session() deletes the sandbox even when sandbox.aclose() raises — the container and its reserved port must never leak. Worker (runtime/server/worker/process.py): - shutdown drain is bounded (2s): a wedged outbound pipe would hang the queue join forever. - cancel enqueues its Cancelled frame synchronously instead of via an untracked create_task that GC could drop. Adversarial-review fixes on top of the port: - the stale-handle teardown and close() use socketio's shutdown(), not disconnect() — disconnect() is a library no-op on a dropped transport and never stops the background reconnect task, so the leak/double- socket the teardown targets persisted. - submit_http_call attaches its cache+emit callback at submission (not in the timeout branch), so the calls-pop/pending-store transition is in the same completion-callback batch on the HTTP path too; the HTTP-answered path self-acks its cached copy. - the HTTP fast path is opportunistic: transport/status failures fall back to SIO instead of leaking raw httpx errors through remote() and try_remote()'s Result contract. - docs/concepts/remote-calls.mdx + try_remote docstring updated for CallCancelled; client.py __all__ and test_public_exports extended. Docs: PROTOCOL.md documents resume/ack, the no-silent-loss invariant, and the CallCancelled/WorkerExited client mapping. NOT ported (deliberately): the single-transport migration (POST /call removal), reconnection-knob removal, boot_error cleanup, /log rewrite, codec ext-type removal, and every master test the branch's merge-base lag would have deleted (test_extract, test_swe_env, docker retry, client options — all phantom deletions). Co-authored-by: FatPigeorz <wjhhhhhha0@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth slice of the #122 split (stage map in the coordination comment; A = #133, B = #134, C = #135). Ports the "bucket A" core-reliability subset of the source branch's reliability refactor (b81f2fc) onto master — no wire-contract removals: the HTTP
/callfast path, reconnection knobs, structured/logbridge, and codec ext types all stay (those belong to later stages).What's in
Definitive resume (server) —
resumenow resolves every id: cached result → replayed; still running → left alone; evicted/unknown →call:error type=ResultUnavailableinstead of silence that hung the host'sremote()forever. The completed-task transition is atomic w.r.t. resume: results are cached synchronously inside the completion callback batch (same loop turn as thecallspop), on both the SIO and HTTP submission paths — a window the source branch itself had, where a resume racing a completion misfired ResultUnavailable for a live result.Task-retention fixes — result-emit tasks are strongly referenced (
emit_tasks); the worker's cancel frame is enqueued synchronously; the worker shutdown drain is bounded (2s) so a wedged outbound pipe can't hang shutdown.Client teardown — a disconnected stale SIO handle is
shutdown()(notdisconnect(), which is a library no-op on a dropped transport and never stops the reconnect loop) before a fresh client is built;close()gets the same treatment. Without this, every transport flap leaked an auto-reconnecting client + aiohttp session and eventually a second live/rpcsocket.Typed failure vocabulary — server-side cancellation raises
CallCancelled(aRemoteCallErrorsubclass), not a bareasyncio.CancelledErrorthat reads as local task cancellation (intentional breaking change). Newtry_remote() -> Result[R](Ok | Failed) onRuntimeClientandSandboxfor harnesses that branch on outcomes at scale. The HTTP fast path is now opportunistic — transport failures fall back to SIO instead of leaking raw httpx errors throughtry_remote()'s contract. A malformedcall:errorresolves to a typedMalformedError, not aKeyError.Provider contract —
session()deletes the sandbox even whenaclose()raises (the container and its reserved port must never leak).Docs — PROTOCOL.md documents
resume/ack, the no-silent-loss invariant, and the client error mapping;docs/concepts/remote-calls.mdxupdated.Deliberately NOT ported
The single-transport migration (
POST /callremoval), reconnection-knob removal,boot_errorcleanup, the/logrewrite, codec ext-type removal — and every "deletion" that was actually merge-base lag (the source branch forked before #117/#126–#130/#132:test_extract,test_swe_env, docker-retry tests etc. all survive untouched).Verification
uv run pytest tests/ plugins/— 506 passed (10 new tests: ResultUnavailable + running-call resume, try_remote Ok/Failed, stale-handle shutdown ×2, fast-path fallback, CallCancelled mapping, MalformedError, session-delete contract, export guards). RED evidence captured for the key fixes (each fails against the pre-fix code).uv run ruff check ./uv run pyright— clean / 0 errorsdisconnect()no-op teardown, HTTP-path resume window)🤖 Generated with Claude Code