Replace provider-native web search with a dedicated in-enclave /v1/web_search endpoint - #138
Merged
Merged
Conversation
The `web_search` flag used to switch on whichever native search tool the target model's provider happened to ship: OpenAI's Responses-API `web_search`, Anthropic's `web_search_20250305`, Gemini's `google_search` grounding, or xAI's Responses-API `web_search`. That meant four result shapes, four billable units priced from $0.01 to $0.035 (xAI billed per *citation*), and no search at all on ByteDance, Nous or Z.ai. Now the gateway advertises one ordinary `web_search` function tool and executes it itself, inside the enclave, against Exa: - Works on every text model that can call a function, with identical results, excerpt sizes and citations whichever model is picked. - One flat per-search price on every model, so a client can verify its surcharge as `searches x rate`. - The query leaves the TEE only to Exa; the LLM provider sees nothing beyond the result text in the prompt. The client protocol is untouched. Only `web_search` calls are intercepted — a caller's own tools still come back as `tool_calls` for it to run — so one request still yields one answer. Streaming buffers tool calls unconditionally while search is on (a fragment can't be forwarded before we know whose tool it is) and wraps the rounds in a generator, so the chunk handler sees one flat stream. Progress frames carry a top-level `web_search` object and sources ride out-of-band as `citations`, on the same unsigned terms as generated images. Billing accounts for the loop honestly: the flat surcharge per search that actually reached Exa, plus the tokens of every round, since each round re-sends the conversation and the accumulated results. Failed searches are free. Rounds are capped, and the last one runs with the search tool unbound so a model that would keep searching has to answer. EXA_API_KEY is injected via POST /v1/keys like every provider key; without it the flag is a no-op and /health reports web_search_enabled: false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V9SCMV8LhnpZ21pzBLjzxQ
adambalogh
marked this pull request as ready for review
July 29, 2026 11:52
Replace the in-enclave search tool loop with a plain endpoint the client's
own tool loop calls. The gateway no longer binds a web_search tool, runs
search rounds, or accumulates cross-round usage — chat and completions are
back to stateless request/response, and the web_search request flag is a
deprecated no-op (still accepted and still hashed, so old clients' request
signatures verify unchanged).
- POST /v1/web_search: runs one Exa search inside the enclave, returns
model-ready content plus structured citations, signed with the standard
tee_* fields (request hash over the canonical JSON body, output hash over
content), billed via x402 at the flat WEB_SEARCH_PRICE_USD per search.
Failures (400/502/503) carry no cost block and are never settled.
- OHTTP: the sealed payload's `endpoint` field routes the inner request
("web_search" -> /v1/web_search); absent still means chat, so existing
OHTTP clients are unaffected and relay billing reuses the existing
cost-header/billing-frame channel.
- Removed search_loop.py and the streaming buffering/search-frame machinery
from the chat controller; compute_session_cost no longer takes a search
count (search billing left the chat path entirely).
- web_search.py slims to the Exa client, execution, and formatting; the
tool spec moves to the client.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V9SCMV8LhnpZ21pzBLjzxQ
adambalogh
changed the title
Costs past the cap go unbilled, so the pre-authorization ceiling should sit well above any realistic session's search count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V9SCMV8LhnpZ21pzBLjzxQ
Contributor
There was a problem hiding this comment.
Pull request overview
This PR removes provider-native web search integrations and introduces a dedicated in-enclave POST /v1/web_search endpoint backed by Exa, with OHTTP inner-endpoint dispatch support and flat per-search x402 billing. It keeps the legacy web_search flag in chat/completions requests as a deprecated no-op for wire/signature compatibility.
Changes:
- Added an Exa-backed web search module + controller (
/v1/web_search) with signed responses, citations, and flat-rate cost block generation. - Removed provider-specific “native web search” tool binding/counting and the per-search surcharge from the chat/completions pricing path.
- Extended OHTTP to route sealed inner payloads to either chat (default) or web search via an
endpointdiscriminator.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tee_gateway/web_search.py | New Exa client + execution/formatting logic for in-enclave web search. |
| tee_gateway/controllers/web_search_controller.py | New /v1/web_search handler: request validation, hashing/signing, and optional cost block. |
| tee_gateway/pricing.py | Removes search surcharge from session cost and adds compute_web_search_cost(). |
| tee_gateway/model_registry.py | Replaces per-provider search pricing with a single WEB_SEARCH_PRICE_USD. |
| tee_gateway/llm_backend.py | Removes provider-native search tool logic and wires Exa client configuration into key injection. |
| tee_gateway/controllers/ohttp_controller.py | Adds inner endpoint dispatch to route OHTTP payloads to chat or web search. |
| tee_gateway/controllers/completions_controller.py | Treats web_search flag as deprecated no-op; removes native-search billing/tooling. |
| tee_gateway/controllers/chat_controller.py | Removes native-search binding/counting and search surcharge from chat cost calculation. |
| tee_gateway/definitions.py | Adds a per-session x402 spend cap for /v1/web_search. |
| tee_gateway/config.py | Extends provider config with exa_api_key (not treated as an LLM provider). |
| tee_gateway/main.py | Mounts /v1/web_search, adds x402 route config + health reporting, and updates pricing gate paths. |
| tee_gateway/openapi/openapi.yaml | Marks web_search request flag as deprecated and updates descriptions. |
| tee_gateway/test/test_web_search.py | Replaces old provider-native search tests with endpoint/Exa/pricing/OHTTP dispatch coverage. |
| scripts/run-enclave.sh | Reads EXA_API_KEY from env file and injects it via /v1/keys. |
| README.md | Updates docs to describe the new web-search endpoint and billing semantics. |
| CLAUDE.md | Updates repo architecture/docs to include Exa-backed /v1/web_search behavior and key injection. |
Comments suppressed due to low confidence (3)
tee_gateway/web_search.py:204
- The non-200 response log includes the raw user query (
%r). For the same privacy reasons as above, avoid logging plaintext queries here as well.
logger.warning(
"Exa search failed for %r: HTTP %s %s", query, response.status_code, detail
)
tee_gateway/web_search.py:237
- This info log includes the raw query. If logs are accessible to operators, this is a privacy leak; log a generic message (or only length/hashed query) instead.
logger.info("Exa search for %r returned no results", query)
tee_gateway/web_search.py:253
- The success log includes the raw query (
query=%r). Given the privacy goal of the endpoint, avoid logging plaintext queries; log only counts/latency/size/cost.
logger.info(
"Exa search ok — query=%r results=%d chars=%d reported_cost_usd=%s",
query,
len(citations),
len(content),
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+645
to
660
| if request.path not in ( | ||
| "/v1/chat/completions", | ||
| "/v1/completions", | ||
| "/v1/ohttp", | ||
| "/v1/web_search", | ||
| ): | ||
| return | ||
| try: | ||
| _price_feed.get_price() | ||
| except ValueError as exc: | ||
| logger.warning("Rejecting inference request — price feed unavailable: %s", exc) | ||
| return jsonify({"error": f"Pricing unavailable: {exc}"}), 503 | ||
|
|
||
| if request.path == "/v1/ohttp": | ||
| # OHTTP carries its model inside the ciphertext; web search has no model. | ||
| if request.path in ("/v1/ohttp", "/v1/web_search"): | ||
| return |
Comment on lines
+110
to
+118
| @dataclass(frozen=True) | ||
| class WebSearchOutcome: | ||
| """The result of one web search. | ||
|
|
||
| ``content`` is model-ready text: the client feeds it back to its model as | ||
| the ``web_search`` tool result. ``citations`` are structured so a UI can | ||
| show its sources. ``billable`` is False for failures and for empty/invalid | ||
| calls, so a caller is never charged for a search that produced nothing. | ||
| """ |
Comment on lines
+188
to
+195
| try: | ||
| response = client.post("/search", json=payload) | ||
| except httpx.HTTPError as exc: | ||
| logger.warning("Exa search transport error for %r: %s", query, exc) | ||
| return WebSearchOutcome( | ||
| content=f"Web search error: could not reach the search service ({exc}).", | ||
| is_error=True, | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Removes all four provider-native web search integrations (OpenAI Responses-API
web_search, Anthropicweb_search_20250305, Geminigoogle_searchgrounding, xAI search) and replaces them with a dedicated, model-free endpoint:POST /v1/web_search, backed by Exa and executed inside the enclave.The gateway runs no tool loop of its own. A client whose model asks to search calls this endpoint, feeds the returned
contentback to its model as the tool result, and showscitationsto its user. (The chat-app side is OpenGradient/chat-app#782.)Design
POST /v1/web_search— request{"query", "num_results"?, "recency_days"?}; response carriescontent(model-ready numbered results),citations(structured sources), and the standardtee_*signing fields. Request hash covers the canonical (sorted-keys) JSON body; output hash coverscontent. Mounted viaadd_url_rulelike the other gateway-specific endpoints (/health,/v1/keys,/v1/ohttp).endpointfield routes the sub-dispatch ("web_search"→/v1/web_search); absent still means chat, so existing OHTTP clients are unaffected. Relay billing reuses the existing cost-header channel with zero chat-api changes.WEB_SEARCH_PRICE_USD = $0.015), settled via x402 from the response'sopengradientblock (compute_web_search_cost). Validation failures (400/503) and Exa failures (502) return no cost block and are never settled. A search that ran but matched nothing IS billed. Exa's self-reportedcostDollarsis logged for margin reconciliation only. New x402 route capped at 1 OPG per session (WEB_SEARCH_OPG_SESSION_MAX_SPEND).web_searchflag is a deprecated no-op — still accepted and still part of the signed request hash when sent, so old clients' requests parse and their signatures verify, but it binds nothing and bills nothing.EXA_API_KEYis injected at runtime via/v1/keyslike every provider key; plainhttpxclient, no new dependencies,uv.lockuntouched (PCR-stable).Why an endpoint instead of an in-enclave tool loop
An earlier revision of this branch ran a server-side search loop inside the chat path (intercepting
web_searchtool calls, feeding results back, summing usage across rounds, buffering streamed tool-call fragments). Since the chat-app is the only consumer and it already runs a browser-side tool loop, the loop moved there — deletingsearch_loop.py, the streaming buffering/search-frame machinery, and the cross-round usage accumulation, and keeping the TEE simple.Testing
test_web_search.py: Exa client shaping/formatting/failure modes, flat pricing, endpoint controller (signed shape, billing block, 400/502/503, fail-open on price-feed outage, client-recomputable request hash), and OHTTP inner-endpoint dispatch (routing, chat default, unknown-endpoint rejection).test_ohttp.pyimport error — pytest not installed locally).make lint(ruff + mypy) clean.Deployment notes
EXA_API_KEYmust be added to the enclave host.env; without it/v1/web_searchreturns 503 and/healthreportsweb_search_enabled: false.web_searchflag, old gateways would silently not search; once this gateway deploys, old chat-apps' flag becomes a no-op (they just don't search until updated).