abridge: direct mode — serve @on clients as a standalone HTTP service#131
Merged
Conversation
The tunnel exists for sandboxes with no egress: LLM traffic rides the runtime's Socket.IO connection at the cost of two extra hops and the host process fanning in every rollout's calls. When the sandbox can reach the model-serving network (in-cluster vLLM/SGLang), that detour is pure overhead — serve the same @on(path) handlers next to the engine instead and point the agent's SDK straight at it. - build_app(*clients): one POST route per handler, tunnel-identical contract (JSON-object bodies, ClientResponse out, in-band errors) - build_session_app(factory): per-caller sessions — the API key each agent sends (x-api-key / Bearer) hashes to a session id, the factory builds that caller's client, an LRU caps live clients and acloses evictions. Mint a key per rollout and one server groups them all; agent keys are never forwarded upstream. - agentix-bridge-serve: Anthropic-speaking front for an OpenAI-compatible engine (lazy openai extra, loopback bind default) Multi-backend routing and token capture stay out of scope — that is the full gateway's job; this is the tunnel without the tunnel, and the tunnel remains the mode for egress-less sandboxes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adversarial review of the first cut confirmed three lifecycle holes and one trust-model gap; all fixed: - eviction is now in-flight-safe: the session table stays LRU-bounded, but an evicted client closes only when its last live request releases (previously a burst of fresh caller keys aclosed clients mid-upstream-call, turning valid LLM calls into 502s — also removes the eviction-thrash DoS lever) - a FastAPI lifespan closes the route-probe client at startup and drains every remaining session client at shutdown, matching the cleanup guarantee the tunnel's Proxy.session() gives - the client factory runs outside the table lock via asyncio.to_thread (SDK construction is ~30ms of synchronous SSL/pool setup that was stalling the whole event loop under the lock) - opt-in caller verification: build_session_app(verify_key=...) / --require-key-prefix gates requests to keys the harness minted (401 otherwise); trust model documented in the module and README, and the README example now shows the non-loopback bind it implies - session_id_for() is public and documented so the minting side can correlate upstream x-session-id values; build_app's docstring now states honestly that AbridgeError statuses survive here while the tunnel wire collapses them to 502 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The tunnel exists for sandboxes with no egress: LLM traffic rides the runtime's Socket.IO connection, at the cost of two extra hops and the host process fanning in every concurrent rollout's calls. When the sandbox can reach the model-serving network (in-cluster vLLM/SGLang), that detour is pure overhead — the RL-at-scale scenario wants agents talking to the engine directly, with translation and identity stamping next to the engine instead of on the host.
What
agentix.bridge.serve— "the tunnel without the tunnel":build_app(*clients)— one POST route per@on(path)handler; tunnel-shaped contract (JSON-object bodies,ClientResponseout, in-band errors;AbridgeErrorstatuses survive here, which the tunnel wire can't do).build_session_app(factory)— one client per caller identity: each request's API key (x-api-key/Bearer) maps tosession_id_for(key)(public, so the minting side correlates),factory(session_id)builds that caller's client. LRU-bounded with in-flight-safe eviction (an evicted client closes only after its live requests drain), full cleanup via lifespan at shutdown, factory runs off-loop.agentix-bridge-serve— Anthropic-speaking front for an OpenAI-compatible engine. Mint a key per rollout (the placeholder you already inject) and one server groups them all; agent keys are never forwarded upstream. Opt-in gating with--require-key-prefix <secret>(401 otherwise); loopback bind by default; trust model documented.Deliberately not in scope: multi-backend routing and token capture — that's the full gateway's territory (#122); this PR only removes the host from the data path for the direct-connect case, and the tunnel remains the universal fallback.
Verification
x-session-ids with the raw keys never reaching upstream headers./_health200; bad key → 401; minted key with upstream down → clean in-band 502.Notes for review
This intentionally overlaps in purpose (not code) with #122's SessionForward — it's the minimal direct-connect enabler using only existing clients, meant to be subsumed by the full gateway when that lands. Happy to adjust naming/scope to whatever #122 settles on.
🤖 Generated with Claude Code