Add cancellation and deadline support to Session.process_request - #994
Draft
bmehta001 wants to merge 2 commits into
Draft
Add cancellation and deadline support to Session.process_request#994bmehta001 wants to merge 2 commits into
bmehta001 wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds cross-language cancellation and deadline support for synchronous inference, including engine-level interruption and teardown handling.
Changes:
- Adds native session cancellation, request deadlines, and timeout errors.
- Exposes cancellation/timeouts through C++, Python, C#, and JavaScript.
- Adds chat cancellation and deadline regression tests.
Reviewed changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated 16 comments.
Show a summary per file
| File | Description |
|---|---|
sdk_v2/python/src/foundry_local_sdk/session.py |
Adds timeout, cancellation, and teardown signaling. |
sdk_v2/python/src/foundry_local_sdk/request.py |
Adds request timeout configuration. |
sdk_v2/python/src/foundry_local_sdk/_native/build_cffi.py |
Extends Python C ABI definitions. |
sdk_v2/js/src/session.ts |
Adds AbortSignal cancellation and session cancellation. |
sdk_v2/js/src/request.ts |
Adds request deadlines. |
sdk_v2/js/src/detail/native.ts |
Extends native TypeScript interfaces. |
sdk_v2/js/src/detail/errors.ts |
Adds the timeout error code. |
sdk_v2/js/native/src/session.h |
Declares native cancellation methods. |
sdk_v2/js/native/src/session.cc |
Implements native session cancellation bindings. |
sdk_v2/js/native/src/request.h |
Declares native timeout support. |
sdk_v2/js/native/src/request.cc |
Implements native timeout binding. |
sdk_v2/cs/src/Session.cs |
Adds token/session cancellation and teardown signaling. |
sdk_v2/cs/src/Request.cs |
Adds TimeSpan deadlines. |
sdk_v2/cs/src/Detail/NativeMethods.cs |
Extends C# ABI declarations. |
sdk_v2/cs/src/Detail/FoundryLocalApi.cs |
Maps timeout errors and session cancellation. |
sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc |
Adds cancellation and deadline tests. |
sdk_v2/cpp/src/inferencing/session/session.h |
Defines session cancellation state and watchdog support. |
sdk_v2/cpp/src/inferencing/session/session.cc |
Implements cancellation, tracking, and deadlines. |
sdk_v2/cpp/src/inferencing/session/session_manager.h |
Documents process-wide cancellation. |
sdk_v2/cpp/src/inferencing/session/session_manager.cc |
Cancels sessions during shutdown. |
sdk_v2/cpp/src/inferencing/session/request.h |
Implements reusable request deadlines. |
sdk_v2/cpp/src/inferencing/session/oga_generator_cancellable.h |
Declares the OGA cancellation adapter. |
sdk_v2/cpp/src/inferencing/session/oga_generator_cancellable.cc |
Implements engine termination. |
sdk_v2/cpp/src/inferencing/session/live_session_registry.h |
Declares live-session tracking. |
sdk_v2/cpp/src/inferencing/session/live_session_registry.cc |
Implements the live-session registry. |
sdk_v2/cpp/src/inferencing/session/cancellable.h |
Defines the cancellation interface. |
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.h |
Adds request-aware embedding generation. |
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc |
Adds embedding cancellation checks. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc |
Publishes chat generators for cancellation. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h |
Makes chat generators cancellable. |
sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc |
Adds cancellation throughout audio generation. |
sdk_v2/cpp/src/inferencing/generative/audio/audio_generator.h |
Makes audio generators cancellable. |
sdk_v2/cpp/src/c_api.cc |
Implements new C ABI operations. |
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h |
Implements C++ wrapper methods. |
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h |
Exposes C++ cancellation and deadlines. |
sdk_v2/cpp/include/foundry_local/foundry_local_c.h |
Extends the public C ABI. |
sdk_v2/cpp/CMakeLists.txt |
Builds the new cancellation sources. |
Suppressed comments (2)
sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc:604
- This second raw OGA decode loop has the same termination-exception gap:
Session::Cancel()now terminates the generator duringGenerateNextToken(), but this helper does not catch the expected runtime error. A Nemotron cancellation or deadline can therefore bypass normal cancellation/timeout handling. Handle termination whenoriginal_request.ShouldStop()is true and rethrow unrelated engine failures.
while (!generator.IsDone() && !generator.IsSessionTerminated() && !original_request.ShouldStop()) {
sdk_v2/js/src/session.ts:275
Request_Cancelonly sets the request flag; it never invokes the active generator'sCancel(). Consequently this AbortSignal still waits for an in-progress prefill/decode call and cannot interrupt a non-terminating compute as documented. Moreover, the native cancellation path returns aFINISH_NONEresponse rather thanFOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, so this promise resolves instead of rejecting. Associate request cancellation with its active generator and define the abort rejection before exposing this option.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| /// On expiry the in-flight generation is interrupted — including mid-compute, not just | ||
| /// at token boundaries — the session's model reference is released, and | ||
| /// Session_ProcessRequest returns FOUNDRY_LOCAL_ERROR_TIMEOUT. | ||
| FL_API_STATUS(Request_SetTimeoutMs, _In_ flRequest* request, uint64_t timeout_ms); |
Comment on lines
+130
to
+131
| generators = cancel_state_->active_generators; | ||
| requests = cancel_state_->active_requests_list; |
Comment on lines
+183
to
+184
| const bool woken = cancel_state_->cv.wait_for(lock, timeout, [this] { | ||
| return cancel_state_->active_requests == 0 || cancel_state_->cancel_requested; |
Comment on lines
+215
to
+228
| // A session cancelled during teardown must not start new work — otherwise a caller | ||
| // looping over requests could keep the model refcount pinned past Manager::Shutdown. | ||
| { | ||
| std::lock_guard<std::mutex> cancel_lock(cancel_state_->mutex); | ||
| if (cancel_state_->cancel_requested) { | ||
| FL_THROW(FOUNDRY_LOCAL_ERROR_INVALID_USAGE, "session has been cancelled"); | ||
| } | ||
|
|
||
| ++cancel_state_->active_requests; | ||
| cancel_state_->active_requests_list.push_back(&request); | ||
| } | ||
|
|
||
| // Start the wall-clock budget (if any) and clear stale stop state from a prior run. | ||
| request.ArmDeadline(); |
| // | ||
| // Snapshot first and cancel outside any lock — Session::Cancel() reaches into the ORT | ||
| // GenAI engine, and a cancelled session unwinding calls back into Deregister(). | ||
| std::vector<Session*> to_cancel = LiveSessionRegistry::Instance().Snapshot(); |
Comment on lines
+410
to
+412
| // Reusing the object must not leave it permanently in a timed-out state. | ||
| Request request2; | ||
| request2.AddOwnedItem(MakeMessage(FOUNDRY_LOCAL_ROLE_USER, "What is 2+2? Answer with just the number.")); |
Comment on lines
+278
to
+280
| if (request.timed_out.load(std::memory_order_relaxed)) { | ||
| FL_THROW(FOUNDRY_LOCAL_ERROR_TIMEOUT, "request timed out after ", request.Timeout().count(), "ms"); | ||
| } |
Comment on lines
+110
to
+112
| const double ms = info[0].As<Napi::Number>().DoubleValue(); | ||
| return CallChecked<Napi::Value>(env, [&]() -> Napi::Value { | ||
| impl_->SetTimeout(std::chrono::milliseconds(ms > 0 ? static_cast<int64_t>(ms) : 0)); |
| return MakeStatus(FOUNDRY_LOCAL_ERROR_INVALID_ARGUMENT, "null argument"); | ||
| } | ||
|
|
||
| AsImpl(request)->SetTimeout(std::chrono::milliseconds(timeout_ms)); |
Comment on lines
+269
to
+271
| if (signal?.aborted) { | ||
| request.cancel(); | ||
| } |
bmehta001
marked this pull request as draft
August 12, 2026 18:03
Session.process_request() previously had no way to time out or be cancelled outside the streaming path, so a non-terminating non-streaming generation permanently pinned the session refcount. This caused Model.Unload() to fail with '1 session(s) still using it', FoundryLocalManager.close() to exceed its drain deadline, and process teardown to intermittently crash. C++ core: - Add ICancellable interface and OgaGeneratorCancellable adapter so Session can interrupt ORT GenAI mid-compute (SetRuntimeOption(terminate_session)), not just between token boundaries. - Add Request::SetTimeout/ArmDeadline/ShouldStop for a re-armable wall-clock deadline that applies to streaming and non-streaming requests alike. - Add Session::Cancel() with active-generator tracking and a deadline watchdog thread; wire it into all chat/audio/embeddings generation loops. - Cancel() now latches request.canceled on every in-flight request (not just published generators) so finish_reason and history rollback are correct even when cancellation lands during prefill. - Add LiveSessionRegistry so SessionManager::CancelAll() can reach every live session, including ones created via the direct API that never took a SessionRegistration. - Add FOUNDRY_LOCAL_ERROR_TIMEOUT and two C ABI vtable entries: Request_SetTimeoutMs, Session_Cancel (appended to preserve ABI ordering). Bindings (C++ wrapper, Python, C#, JS): - Request.SetTimeout()/set_timeout()/setTimeout(), Session.Cancel()/cancel(). - Non-streaming ProcessRequestAsync (C#) and processRequest (JS) now accept a CancellationToken / AbortSignal that genuinely interrupts an in-flight generation, not just prevents scheduling. - Session teardown (Python _close, C# Dispose, JS dispose) now cancels the session itself, covering the non-streaming case that was previously invisible to shutdown. Tests: added ChatSessionTest coverage for timeout enforcement, timeout error code, deadline re-arming across reused requests, mid-flight cross-thread cancellation, cancel-before-request rejection, and cancel-when-idle safety. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 86c9e761-c751-48bc-8e73-2db5257c9e88
Prevent deadline enforcement from poisoning reusable chat state, preserve timeout semantics when ORT terminates raw generators, cancel inference before joining web workers, and release JS streaming callbacks on every exit. Files changed: - sdk_v2/cpp/src/inferencing/generative/audio/audio_session.cc - sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc - sdk_v2/cpp/src/inferencing/generative/chat/chat_session.h - sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc - sdk_v2/cpp/src/inferencing/session/oga_generator_cancellable.h - sdk_v2/cpp/src/inferencing/session/session.cc - sdk_v2/cpp/src/inferencing/session/session.h - sdk_v2/cpp/src/inferencing/session/session_manager.cc - sdk_v2/cpp/src/manager.cc - sdk_v2/cpp/src/service/audio_transcriptions_handler.cc - sdk_v2/cpp/src/service/chat_completions_handler.cc - sdk_v2/cpp/src/service/responses_handler.cc - sdk_v2/cpp/test/CMakeLists.txt - sdk_v2/cpp/test/internal_api/chat/chat_session_test.cc - sdk_v2/cpp/test/internal_api/oga_generator_cancellable_test.cc - sdk_v2/cpp/test/internal_api/session_manager_test.cc - sdk_v2/js/native/src/session.cc - sdk_v2/js/test/streaming.test.ts Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 16cc8b6f-2bc4-4469-bf59-f21439569349
bmehta001
force-pushed
the
bhamehta001/support-cancellation
branch
from
August 13, 2026 18:19
6b1dab0 to
3874896
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Session::process_request()exposed no cancellation, timeout, or deadline mechanism.Request.cancel()was only wired into the streaming path, so a non-terminating non-streaming generation could permanently pin the session refcount.Consequences:
Model.Unload()failed with "1 session(s) still using it".FoundryLocalManager.close()exceeded its fixed 10-second drain deadline and left the model loaded.1073740791(0xC0000409).Fix
C++ core
ICancellableinterface andOgaGeneratorCancellableadapter soSessioncan interrupt ORT GenAI mid-compute (viaSetRuntimeOption("terminate_session", "1")), not just between token boundaries.Request::SetTimeout/ArmDeadline/ShouldStop— a re-armable wall-clock deadline that applies to streaming and non-streaming requests alike.Session::Cancel()with active-generator tracking and a deadline watchdog thread; wired into every chat/audio/embeddings generation loop.Cancel()now latchesrequest.canceledon every in-flight request (not just published generators), sofinish_reasonand history rollback are correct even when cancellation lands during prefill (before any generator is published).LiveSessionRegistrysoSessionManager::CancelAll()reaches every live session, including ones created via the direct API that never took aSessionRegistration— this is what fixes manager teardown and the 0xC0000409 crash.FOUNDRY_LOCAL_ERROR_TIMEOUTand two new C ABI vtable entries (Request_SetTimeoutMs,Session_Cancel), appended to preserve ABI ordering.Bindings (C++ wrapper, Python, C#, JS)
Request.SetTimeout()/set_timeout()/setTimeout(),Session.Cancel()/cancel().ProcessRequestAsync(C#) andprocessRequest(JS) now accept aCancellationToken/AbortSignalthat genuinely interrupts an in-flight generation (registered against the native cancel), not just prevents scheduling._close, C#Dispose, JSdispose) now cancels the session itself, covering the non-streaming case that was previously invisible to shutdown.Testing
Added
ChatSessionTestcoverage:FOUNDRY_LOCAL_ERROR_TIMEOUT.Request.Session::Cancel()stops an in-flight non-streaming request promptly.Cancel()before a request rejects immediately (no unbounded generation).Cancel()is idempotent and safe when idle.Full C++ unit suite: 1065/1069 passed (4 skips are pre-existing/unrelated audio integration tests requiring models not present in this environment).
Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com