abridge: configurable tunnel request window + explicit SDK retry policy#152
Merged
Conversation
The tunnel answered every agent request within a hard-coded 600s window (_start_tunnel already took request_timeout; Proxy.start never passed it). The window's timer starts when the request reaches the sandbox — strictly before the host's upstream call — so any upstream call approaching 600s lost the race: the agent got a 504 while the host handler kept running, and the SDK's silent default max_retries=2 let the handler occupy up to ~3x the window on a result nobody was waiting for. Long-generation models hit this in practice; the only workaround was capping the model's time budget to fit the infra. - Proxy(*clients, request_timeout=600.0) threads the window into the in-sandbox tunnel. Sizing rule documented as a LOWER bound: client timeout x (1 + max_retries), with SDK backoff and Retry-After waits on top — leave real margin. Non-positive values rejected. - AnthropicFromOpenAIClient, OpenAIClient, and AnthropicClient take max_retries and default it to 0 — retry policy is the operator's call, never a hidden multiplier behind the tunnel window. agentix-bridge-serve grows --upstream-max-retries so CLI operators can opt back in. - Forward's default deadline drops to 540s, strictly under the default window it races (an equal deadline always loses — the tunnel timer starts first). - Tests include a real end-to-end tunnel: a never-answering host 504s at ~request_timeout, so a regression re-hardcoding the window during route registration blows the time budget instead of staying green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
abridge's credential-isolation tunnel forwards each agent request from an in-sandbox loopback FastAPI, over Socket.IO, to the host-side
Proxyclient that calls the real upstream. The per-request wait window was hard-coded 600s (_start_tunneltook arequest_timeoutparam;Proxy.startnever passed it). That window's timer starts when the request reaches the sandbox — strictly before the host's upstream call begins — so any upstream call approaching 600s lost the race: the agent got a504 tunnel timed outwhile the host handler kept running, and the openai/anthropic SDK's silent defaultmax_retries=2let the handler occupy up to ~3× the window on a result nobody was waiting for. Long-generation models hit this in practice; the only workaround was capping the model's own time budget to fit the infra.What
Proxy(*clients, request_timeout=600.0)threads the window into the in-sandbox tunnel. The sizing rule is documented as a lower bound: clienttimeout × (1 + max_retries), with SDK backoff andRetry-Afterwaits on top — leave real margin. Non-positive values rejected.AnthropicFromOpenAIClient,OpenAIClient,AnthropicClientgainmax_retries, default0— retry policy is the operator's call, never a hidden multiplier behind the window.agentix-bridge-servegrows--upstream-max-retriesso CLI operators can opt back in.Forward's default deadline drops to 540s, strictly under the default window it races (an equal deadline always loses — the tunnel timer starts first).Tests
Includes a real end-to-end tunnel: a never-answering host returns 504 at ~
request_timeout, so a regression that re-hardcodes the window during route registration blows the time budget instead of staying green. Plus the threading, default, validation, and per-client retry-policy assertions.Reviewed by a second co-author (Codex) + adversarial pass; confirmed findings folded in (AnthropicClient retry knob, serve.py opt-in, Forward default, lower-bound wording, real-tunnel test).
🤖 Generated with Claude Code