tito: token-in-token-out session-recording gateway (PR #122 stage B)#134
Merged
Conversation
Port plugins/tito from origin/abridge/gateway-tito onto master as the second slice of the #122 split: a self-contained workspace member providing the TITO gateway — a FastAPI session server that keeps a token-aligned trajectory per session and proxies chat completions across a pool of OpenAI-compatible sglang replicas, with a native engine (incremental pretokenization, append-only message validation, single-step rollback, mismatch reporting). No vendored Miles; imports nothing from agentix core beyond the namespace. Planned fixes on top of the port: - BackendPool: a downed replica now recovers — report_up is wired into the proxy success path, and a down mark expires after down_cooldown seconds (half-open) so a replica that is never picked isn't blacklisted forever. - BackendPool: sticky session pins are LRU-bounded (max_pins) — the recommended rollout flow never DELETEs its session (the trajectory must survive for harvest), so the pin map grew without bound. - chat route: force stream=false upstream (drop stream_options) — the TITO flow needs the full JSON completion; a stream:true agent previously got a 500 from json-parsing the SSE bytes. Fixes from the pre-PR adversarial review: - chat route: accept tool-call-only assistant turns with content:null (sglang/vLLM emit them routinely) — previously the first tool call of an agentic rollout 502'd. - trust_remote_code is now an explicit opt-in (config field + --trust-remote-code), never a hardcoded True: loading a tokenizer executes Python shipped inside the checkpoint repo. - trajectory: keep only the two reachable checkpoints (single-step rollback) instead of every full prefix+completion token list — O(turns^2) dead memory per session. - chat route: malformed request JSON is a 400; structurally malformed 200 upstream bodies are a clean 502 (not an unhandled 500). - proxy responses pass the upstream body through verbatim — no JSON re-encode (which 500'd on NaN/Infinity and perturbed byte-sensitive bodies). - prefix-mismatch diagnostic no longer raises a bare ValueError when the new sequence is shorter than the stored checkpoint. - CLI: --backend-url is repeatable (multi-replica pool) and --routing-policy is exposed — the pool was Python-API-only. - packaging: requires-python >=3.11 (StrEnum), drop unused setproctitle dep, sglang-specific docs corrected. - concurrency (from the concurrency review pass): a retry chain that outruns the trimmed checkpoint window now re-renders from scratch instead of merging onto an empty prefix (which silently dropped the whole stored history and recorded a corrupt trajectory); rollback is planned+validated before any mutation, so a rejected (400) request no longer commits truncation side effects; the phase-3 interference guard uses a monotonic version counter (num_assistant was ABA-prone); DELETE sets `closing` inside the lock (a cancelled DELETE no longer wedges the session); the request body is read before taking the session lock (a dribbling upload no longer blocks DELETE); the forget-on-delete middleware matches only the exact session resource. Tests: real HTTP-surface coverage (httpx ASGITransport over the actual FastAPI app: session flow, forced non-streaming, tool-call null content, failover + cooldown recovery, error shapes, verbatim passthrough) on top of the ported unit tests. plugins/tito wired into the uv workspace + pyright include/extraPaths. Co-authored-by: FatPigeorz <wjhhhhhha0@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of the #122 split (stage map in the coordination comment; stage A was #133). Ports
plugins/titofromorigin/abridge/gateway-titoonto master as a self-contained uv workspace member, plus fixes from the planned list and two adversarial review passes.What's in
The TITO gateway — a FastAPI session server that keeps a token-aligned trajectory per session and proxies chat completions across a pool of OpenAI-compatible sglang replicas. The native engine does incremental pretokenization (merge onto the stored token prefix instead of re-rendering), append-only message validation, single-step rollback on agent retries, and from-scratch-vs-accumulated mismatch reporting. No vendored Miles; imports nothing from agentix core beyond the namespace. Server side of the Stage A
SessionForwardcontract (POST /sessions,POST /sessions/{id}/v1/chat/completions, harvest viaGET, reap viaDELETE).Planned fixes (from the #122 review)
report_upwired into the proxy success path + down marks expire afterdown_cooldown(half-open) — a downed replica was previously blacklisted foreverstream=falseupstream (the TITO flow needs the full JSON completion with logprobs)max_pinsAdversarial review fixes (2 passes, 3 perspectives + 2 rebutters per finding)
content:nullno longer 502 — previously the first tool call of an agentic rollout died (sglang/vLLM emit null content on tool-call-only turns)trust_remote_codeis an explicit opt-in (config +--trust-remote-code), was a hardcodedTrueversion(rollback+update restorednum_assistant, admitting stale updates)closingset inside the lock (a cancelled DELETE no longer wedges the session forever); request body read before the lock (a dribbling upload no longer blocks DELETE)--backend-urlrepeatable +--routing-policy(the multi-replica pool was Python-API-only); packaging:requires-python >=3.11, unusedsetproctitledroppedVerification
uv run pytest tests/ plugins/— 486 passed (26 new tests: real HTTP surface viahttpx.ASGITransport— session flow, forced non-streaming, tool-call null content, failover + cooldown recovery, error shapes, lock/cancellation behavior — plus pool + engine regression tests)uv run ruff check ./uv run pyright— clean / 0 errorsgit diff origin/abridge/gateway-tito -- plugins/tito)Deferred (disputed in review, engine semantics for follow-up):
TEMPLATE_RELEVANT_KEYSomittingname/tool_call_idin default-mode append-only matching.🤖 Generated with Claude Code