fix(midnight-liquidation): add discovery cooldown breaker and rate cap

[BZahory]

Summary

Incident hardening for CRTR-2857 (Sev 1, 2026-07-25): the midnight-liquidation bot's discovery
traffic drove a sustained 429 retry storm against the Blue API GraphQL edge (WAF rule
RateLimitMidnightLiquidationBot, ~33 rps global budget shared across all bot instances via the
candidates endpoint's server-side fan-out). ~91% of requests were WAF-blocked for 5+ hours and the
bot went effectively blind.

The structural gap: fetchWithRetry honors a 429's Retry-After within one discovery pass,
but the runner re-fires a full scan (up to MAX_DISCOVERY_PAGES = 100 pages, include_matured
always on) every ~2s block with no memory that the last pass was throttled. Under a throttle, that
re-hammering is the outage.

Two bot-side defenses, both env-tunable:

1. Cross-tick discovery circuit breaker (withDiscoveryCooldown): a failed discovery pass
   latches discovery off for min(max, max(serverRetryAfter, base·2^(failures−1))) — seeded from
   the server's own Retry-After when it sent one (surfaced via the new
   HttpRetryExhaustedError in @repo/utils), with the exponential ramp as a floor so a
   degenerate short header can't defeat the breaker. Latched ticks log discover.cooldown and run
   with zero candidates — which the tick already tolerates (discovery is a coverage source, never a
   correctness dependency). The first successful pass resets the latch
   (discover.cooldown_reset). Defaults: base 60s, cap 600s (above the WAF's own
   Retry-After: 300).

2. Client-side rate cap on candidate-page fetches: a token bucket (createTokenBucket, the
   same idiom as the venue-quoting HTTP_RPS limiter) gates every page request, retries included.
   This is load-bearing, not belt-and-braces: the candidates endpoint sits behind CloudFront with
   Cache-Control: public, max-age=2, stale-while-revalidate=1 (verified live), so a throttled
   origin can be masked as a 200 served stale — in which case no error ever surfaces and the
   breaker alone would never trip. Defaults: 2 rps, burst 4 — conservative because the WAF counter
   is CONSTANT (one global budget shared by all instances, amplified by the server-side GraphQL
   fan-out).

Fail-safe reasoning

During a throttle the bot is already ~90%+ blind, so backing off costs approximately nothing in
coverage, lets the origin recover, and discovery auto-resumes when the window expires. Backing off
a throttled endpoint is strictly correct. Failures still propagate to the tick's existing
discover.error path, and pending-queue maintenance is unaffected (it runs outside the tick).

New env knobs (all optional)

Var	Default	Meaning
DISCOVERY_COOLDOWN_BASE_MS	60000	First-failure cooldown; exponential base. 0 disables the exponential floor (server Retry-After still honored).
DISCOVERY_COOLDOWN_MAX_MS	600000	Cooldown cap; must be ≥ base (validated fail-loud).
DISCOVERY_HTTP_RPS	2	Token-bucket refill for candidate-page requests.
DISCOVERY_HTTP_BURST	4	Token-bucket burst capacity.

@repo/utils change

fetchWithRetry now throws HttpRetryExhaustedError (carrying status + parsed retryAfterMs)
when the retry budget is exhausted on a 429/5xx. The message is byte-identical to the previous
plain Error, and all existing consumers (blue-liquidation, monitor-bot) assert on messages only —
verified by the full suite.

Root-cause attribution caveat

The WAF evidence keys on "the midnight bot's auth string", but this bot sends no auth — it
calls the public REST candidates endpoint, and the keyed GraphQL traffic comes from that endpoint's
server-side fan-out. Per the integration team, the legacy Blue/Midnight bots (running in parallel)
DO hold API keys, so the keyed volume may be partly or wholly theirs rather than this bot's.
Distinguishing needs Platform data (WAF source IPs: markets-API backend egress vs legacy-bot
egress). This hardening is correct and low-cost regardless of the split — this bot's cross-tick
discovery volume was unbounded either way.

Out of scope / follow-up

• WAF/rate-limit changes: deliberately untouched — Platform owns the rule, and relaxing it is
  dangerous (12–18× origin step; cf. the 2026-06-27 V8-heap OOM on this key).
• Recommended follow-up: decouple discovery cadence from the 2s block loop (discovery does not
  need to re-scan the full universe every block). Tracked as a separate refactor, not done here.
• Reviewer-suggested follow-ups (deferred to keep the Sev-1 fix minimal): genericize
  withDiscoveryCooldown and hoist it into @repo/bot-kit so blue-liquidation (same
  fetchWithRetry re-hammering exposure) can reuse it; write a TIB capturing the circuit-breaker
  pattern choice and the default calibration against the WAF budget.

Verification

• bun test: 844 pass / 2 pre-existing fork-suite failures (require RPC_URL_8453 +
  anvil; fail identically on a clean tree).
• New tests: breaker latch/skip/resume, exponential ramp + cap, Retry-After seeding + floor,
  reset-on-success; rate-cap slot ordering (retries included); HttpRetryExhaustedError fields +
  message stability; config knob defaults/overrides/validation. Break-one-assertion check done.
• bun run --filter @morpho-org/midnight-liquidation typecheck, --filter @repo/utils typecheck:
  clean. bun lint: 0 warnings. bun format: applied.
• bun run --filter @morpho-org/monitor-bot test (affected via @repo/utils): 222 pass.
• Reviewer-agent pass against CONVENTIONS.md: one must-fix (README operator surface — env-var table
  • discovery section) addressed in this branch; module boundaries, config idioms, log-event naming,
    @repo/utils consumer safety all verified clean.

🤖 Generated with Claude Code