Skip to content

fix(routing): skip known-exhausted accounts at admission#57

Open
iceteaSA wants to merge 1 commit into
cortexkit:mainfrom
iceteaSA:fix/routing-skip-exhausted
Open

fix(routing): skip known-exhausted accounts at admission#57
iceteaSA wants to merge 1 commit into
cortexkit:mainfrom
iceteaSA:fix/routing-skip-exhausted

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Stacked on #54 and #56 — the base merge (145c489) brings isQuotaExhausted (#54) and the getSidebarState(path) overload + snapshot-level quota.checkedAt (#56). Only the top commit (d5f1fdd) is this PR's change; merge those two first and this reduces to one commit.

Why

Rate-limit marks are per-process and in-memory. With several concurrent opencode processes and fallback-first routing pinned at an exhausted account, every process independently pays one doomed admission probe (admission-time usage_limit_reached → mark → retryable error → reroute) before learning what the machine-global sidebar file already knows: the account is at 100% with a reset days away. The operator sees each discovery as a visible retry/error flash.

What

Admission-time candidate selection now consults quota before probing:

  • Dual source with freshness precedence: in-memory QuotaManager peek vs the shared sidebar file row (compared by primary.checkedAt, then snapshot checkedAt, then entry checkedAt). The fresher source is selected — the file wins only when strictly newer, memory wins ties, and an empty in-memory cache (fresh process) defers to a valid file row. isQuotaExhausted (type-safe, fail-open) is applied only to the selected source.
  • Fallback filter: exhausted candidates are dropped after the existing killswitch/rate-limit filters; skipped accounts are never probed, so their backoff/mark state is untouched. Applied to both the proactive (fallback-first) gate and the reactive iterator via a shared memoized selection.
  • Exhausted primary: synthesizes the existing killswitch-style 429 (reason quota-exhausted, Retry-After from the account's own resetsAt) so the reroute happens without the doomed probe — only when a non-exhausted fallback survives.
  • Safety valves: unknown/missing/malformed/past-reset quota is never exhausted (fail-open); if filtering would remove the last admission path, the current wire-probe order is fully restored — a stale or corrupt file can never brick routing, the wire stays the final authority. Sidebar file read at most once per request, tolerant reader, no-throw.
  • Each skip logs on the quota channel: admission skip: exhausted account {accountId, source, resetsAt}.

Verification

  • RED-first: exhausted first fallback (expected client-alt, got work-alt), fresh-process file exhaustion, newer-exhausted-file vs stale-healthy-memory, exhausted primary reroute — all fail pre-fix; 7 fail-open characterizations proven non-vacuous by reverse-applying the src diff (they pass on reverted source, i.e. they pin non-interference).
  • Review (gemini-3.1-pro): APPROVE 0 must / 0 should — adversarial-file surface (block/steer/brick), empty-cache tie-break, filter ordering (no state mutation for skipped accounts), once-per-request memoization, proactive+reactive coverage, and reverse-apply RED all verified.
  • Gates: build ✓ · tsc ✓ · full suite green ✓ · biome clean.

View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


Summary by cubic

Skip probes for accounts already known to be quota‑exhausted by consulting the freshest, identity‑matched quota from memory or the shared sidebar. When main is exhausted and a healthy fallback exists, short‑circuit with a precise 429 using the earliest exhausted window reset.

  • New Features

    • Admission: pick the fresher source (memory vs getSidebarState, once per request), require identity match (ignore mismatched mainAccountId/fallback accountId), drop exhausted candidates, reuse the same selection for proactive/reactive paths; log skips; if filtering would remove the last path, keep the original probe order.
    • Exhausted primary: synthesize a 429 (quota-exhausted) with Retry-After from the earliest exhausted window’s resetsAt; only when a non‑exhausted fallback survives.
    • Quota/state: bind cached snapshots to account identity in QuotaManager (peekFallbackForPolicy(id, identity?) drops on mismatch; setFallback and seeding carry identity). Sidebar/state includes window/snapshot checkedAt, mainAccountId, and per‑fallback accountId, and merges by freshness per account with identity tied to the winning snapshot.
  • Bug Fixes

    • Fail‑open on unknown/malformed usage/reset or past‑reset; an absent window never blocks.
    • Prevent stale overwrites and cross‑identity leaks in setSidebarMachineState; single tolerant sidebar read per request.

Written for commit e8fe6d5. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/opencode/src/index.ts
Comment thread packages/opencode/src/core/refresh-all-quota.ts Outdated
Comment thread packages/opencode/src/index.ts Outdated
Comment thread packages/opencode/src/tests/integration.test.ts
@iceteaSA
iceteaSA force-pushed the fix/routing-skip-exhausted branch from d5f1fdd to 8dfde55 Compare July 22, 2026 10:03
@ualtinok

Copy link
Copy Markdown
Contributor

Thanks. The candidate-loop replay safety work is useful, but this stacked branch needs revision before merge:

  1. Rebase onto current main (v0.4.2) as a clean delta. Do not carry the older background-refresh/session-routing stack or discard authoritative reset propagation from the WebSocket work.
  2. Require main-account identity agreement before treating shared quota as fresh, and pass the identity when storing refreshed main quota. Do the equivalent token-bound check for fallbacks.
  3. Admission exhaustion currently inspects only the primary window. Either evaluate every present quota window, matching policy semantics, or document and test an intentional provider-specific primary-only contract.
  4. Harden cross-identity sidebar merge behavior with a two-writer account-switch test so an older writer cannot replace newer identity/quota state.
  5. Test the real request selector in index.ts, not only the TUI resolver. Also explicitly confirm the intended fail-open behavior when filtering would otherwise remove every route.

Please rerun the full gate on the rebased branch.

@iceteaSA
iceteaSA force-pushed the fix/routing-skip-exhausted branch 2 times, most recently from c03687c to 15d11bb Compare July 22, 2026 20:10
@ualtinok

Copy link
Copy Markdown
Contributor

Thanks for the revision. The clean rebase, main/sidebar identity handling, reset expiry, both-window evaluation, and production-selector coverage are now in place.

One blocker remains: fallback quota is not identity-bound across re-login. getFallback(id, token) invalidates a mismatched token, but persisted account.quota is immediately reseeded under the new token and admission reads it by internal ID without checking the live fallback accountId. A replacement identity can therefore inherit the prior identity's exhausted quota and then write that stale quota under the replacement identity.

Bind fallback cached and persisted quota to the stable fallback identity, then add production-loader regressions for fallback re-login identity isolation and secondary-only admission exhaustion.

@iceteaSA
iceteaSA force-pushed the fix/routing-skip-exhausted branch from 15d11bb to e8fe6d5 Compare July 23, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants