fix(orb): harden the federated peer-intelligence trust boundary (#9147, #9148, #9149, #9150, #9166) by JSONbored · Pull Request #9243 · JSONbored/loopover

JSONbored · 2026-07-27T09:00:47Z

Summary

Five related trust-boundary fixes in the ORB federated peer-intelligence subsystem, bundled together because they touch the same call paths and were tracked as a single audit pass.

orb(federated): the bundle signing key IS the anonymization secret — enrolling a peer hands it the key that de-anonymizes your ORB telemetry #9147 — the federated bundle-signing key was the same secret that anonymizes the always-on ORB telemetry export. Enrolling a peer handed them the key that de-anonymizes this instance's unrelated export. Mints a dedicated orb:federated_signing_secret, and decouples instanceId from the anon secret via a third independent secret, so rotating either no longer breaks the other.
orb(federated): one allowlisted key owns the peer median — no instance dedup, no range checks, no freshness, and an unbounded response read on the dashboard path #9148 — one allowlisted key could mint unlimited Sybil peers and own the fleet peer-median outright: no per-instance dedup, no range checks on peer-supplied numbers, no freshness/replay protection, and an unbounded response read on the maintainer-dashboard's request path. Adds range validation, a bounded freshness/window-match gate, receiver-side MIN_DECIDED enforcement, per-instance dedup (last bundle wins), a persisted replay/rollback watermark, and a per-key Sybil cap. Moves the peer pull off the dashboard's request path onto a new 10-minute federated-peer-sync background queue job, which also finally wires pushFederatedBundle into a real tick (closing the other half of orb(federated): the documented peer rate limit is inert at every real call site, push mode is a no-op configuration, and the ingest comments still claim a fail-open that #9046 removed #9166).
orb(broker): enrollment secrets are append-only — re-enrolling mints a second valid secret and revokes nothing, and self-hosters cannot reach the revoke route at all #9149 — issueOrbEnrollment was a bare INSERT: re-running the install flow after a secret leak minted a second simultaneously-valid secret and revoked nothing; the only revoke path required INTERNAL_JOB_TOKEN, unreachable by a self-hosting maintainer. Adds an opt-in rotate (revoke-then-reissue) and a maintainer-reachable revoke path through the OAuth callback (state=<installationId>:rotate / :revoke), gated by the same verifyInstallationAdmin check the enrollment flow already uses. Surfaces the live-enrollment count on both the secret page and the internal installations list, and sets Cache-Control: no-store on the secret page.
orb(relay): pull-mode drain is scoped by installation, not enrollment — a stale container steals and destructively acks the live one's webhooks, silently #9150 — pull-mode relay drain was scoped by installation_id only, so any valid enrollment secret for an installation could drain and destructively-ack another live enrollment's queued webhooks (blue/green swap, secret rotation, or a rebuilt container). Adds orb_relay_pending.enroll_id, tagged at enqueue time with the same winning-enrollment election the push path already uses ([Bug]: Orb relay forward picks an arbitrary enrollment when multiple enrolled rows exist #1783), and scopes both the SELECT and the ack-DELETE to it. Also makes a second live enrollment for an installation observable via a new counter instead of silent.
orb(federated): the documented peer rate limit is inert at every real call site, push mode is a no-op configuration, and the ingest comments still claim a fail-open that #9046 removed #9166 — the documented federated peer rate limiter was inert at every real call site (the only production caller never supplied a bucket), and two comment blocks still described the ingest gate as fail-open after orb(security): ingest endpoints fail OPEN when their token is unset, and /metrics is unauthenticated #9046 made it fail-closed. Adds a module-level default rate bucket so the limiter always engages, and corrects the stale comments.

Judgment calls

orb(federated): one allowlisted key owns the peer median — no instance dedup, no range checks, no freshness, and an unbounded response read on the dashboard path #9148's Sybil-cap/replay-watermark state is a single JSON blob in system_flags (orb:federated_peer_state), not a new table — a self-hosted federation's peer count is expected to stay small enough (tens, not thousands) that an in-memory scan/cap-check per tick is trivially cheap, and this avoids a migration for what's fundamentally bookkeeping state rather than a queryable relation. MAX_INSTANCES_PER_KEY is a fixed constant (10), not operator-configurable yet.
orb(broker): enrollment secrets are append-only — re-enrolling mints a second valid secret and revokes nothing, and self-hosters cannot reach the revoke route at all #9149's rotate is opt-in, not automatic on every issueOrbEnrollment call — orb(relay): pull-mode drain is scoped by installation, not enrollment — a stale container steals and destructively acks the live one's webhooks, silently #9150 explicitly relies on two live enrollments briefly overlapping during a blue/green swap, so auto-revoking on reissue would have broken that. The OAuth callback's state field is the only channel available to carry a rotate/revoke intent + installation id through GitHub's login/oauth/authorize bounce back to our fixed callback URL (GitHub only echoes code+state, not arbitrary query params).
orb(federated): one allowlisted key owns the peer median — no instance dedup, no range checks, no freshness, and an unbounded response read on the dashboard path #9148's freshness window is 7 days and clock-skew tolerance is 5 minutes — generous relative to a "best-effort background sync" cadence, chosen to bound "replay a stale favorable bundle forever" without over-tightening for a sync that might occasionally miss a tick.

Validation

git diff --check
npm run typecheck (full repo, clean)
npm run db:migrations:check / npm run db:schema-drift:check (both clean; migration 0191_orb_relay_pending_enroll_id.sql is the next contiguous number)
npm run ui:openapi:check / npm run selfhost:env-reference:check (no drift)
Targeted test files for every touched module (federated-import, federated-benchmark, federated-bundle, federated-collector, orb-relay, orb-broker, orb-oauth, orb-onboarding, index/scheduled-jobs, job-dispatch) — 360+ tests passing, including new regression/invariant coverage for every fix above
npm run test:coverage (full, unsharded) — not run in this pass per the task's explicit instruction to keep to directly-relevant test files rather than the full local gate; a maintainer follow-up should run the full gate before merge
npm run test:workers / npm run build:mcp / npm run ui:build — not run, same reason as above

Safety

No secrets, wallets, hotkeys, trust scores, or reward values anywhere in code, comments, or this PR text.
The two auth-adjacent changes (orb(broker): enrollment secrets are append-only — re-enrolling mints a second valid secret and revokes nothing, and self-hosters cannot reach the revoke route at all #9149's OAuth rotate/revoke, orb(relay): pull-mode drain is scoped by installation, not enrollment — a stale container steals and destructively acks the live one's webhooks, silently #9150's enrollment scoping) have negative-path tests (non-admin revoke refused, cross-enrollment ack/drain isolation).
UI Evidence — not applicable, no frontend changes.

Notes

Two pre-existing, unrelated test failures were investigated and confirmed NOT caused by this branch: both were latent on the branch's original fork point and are already fixed on origin/main by #9225 (rate-limit DB-error handling) and an unrelated drizzle-related fix; rebasing onto origin/main before pushing resolved both, and the full targeted suite is green.

Closes #9147
Closes #9148
Closes #9149
Closes #9150
Closes #9166

cloudflare-workers-and-pages · 2026-07-27T09:02:37Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	`c64100c`	Commit Preview URL Branch Preview URL	Jul 27 2026, 09:25 AM

superagent-security · 2026-07-27T09:05:08Z

Superagent didn't find any vulnerabilities or security issues in this PR.

…non secret (#9147) Enrolling a federated peer previously handed them this instance's dedicated anonymization secret (the SAME key that HMACs repo/pr identifiers on the always-on #1255 orb export), since buildFederatedBundle signed with getOrCreateAnonSecret. A peer holding that key could dictionary-invert any repo_hash it observed and forge this instance's x-orb-signature. - Add getOrCreateFederatedSigningSecret: a separate, dedicated system_flags secret used only to sign federated bundles. - Decouple instanceId from the anon secret via getOrCreateInstanceIdentitySecret, a third independent secret seeded from the current anon secret on first use (so an upgrading instance keeps its existing instanceId) but stored in its own row from then on, so rotating the anon secret no longer resets the export watermark or the federated peer-median's per-instance accounting. - Correct anonymize.ts's key-separation docstring, which was false for any federated-enabled instance.

…gest comments (#9166) The documented per-collector rate limit (6/min) was inert at every real call site: rateLimitAllows treated a caller-omitted bucket as "unlimited" instead of falling back to a real bucket, and the only production caller (buildFederatedBenchmark) never supplied one. evaluateLocalRateLimit is also pure (read-only), so nothing was ever advancing a bucket's count either. - Add module-level default push/pull buckets in federated-collector.ts; rateLimitAllows now always consults (and advances) a real bucket, whether caller-supplied or the module default. - Bound pullPeerBundles' response read with the same reader /v1/orb/ingest uses (generalized readOrbIngestBody to accept any body-bearing source, so it works for both a Request and a collector Response), and cap the parsed array before any per-element shape check. - Remove two stale "FAIL-OPEN by default" comment blocks in routes.ts left over from #9046, which made isAuthorizedIngest's ingest gate (now fail-closed) read backwards from what the code actually does. pushFederatedBundle wiring into a real tick lands with the #9148 commit, which introduces the background refresh job this naturally slots into.

orb_relay_pending was scoped only by installation_id, so any enrollment secret valid for an installation could drain and destructively-ack another live enrollment's queued webhooks -- a stale re-enrolled container silently stealing a fresh one's events (blue/green swap, secret rotation, or a rebuilt container issued a fresh secret while the old one is still inside its drain timer). - Add orb_relay_pending.enroll_id, tagged at enqueue time with the SAME winning enrollment forwardOrbEvent already elects for the push path (#1783's deterministic ORDER BY). NULL for config_push rows (installation- wide, not consumer-specific) and for any pre-#9150 row, so an untagged row is never orphaned -- pullRelayPending's WHERE matches enroll_id = ? OR enroll_id IS NULL. - forwardOrbEvent now fetches every live enrollment row for an installation (not just the elected winner) so a second live enrollment is observable rather than silent, and increments a new counter, loopover_orb_relay_multiple_live_enrollments_total, whenever more than one exists. - pullRelayPending/enqueueRelayPending both accept an optional enrollId to scope the SELECT and the ack-DELETE together, closing the hole where one container could ack another's rows.

One allowlisted key could mint unlimited Sybil peers and own the median outright: verifyFederatedBundle only checked whether SOME allowlisted key signed a bundle, never bound a key to an instanceId, and importPeerBundles had no per-instance dedup, no range checks, no freshness/replay protection, and pulled peer bundles synchronously on every maintainer dashboard load. - Range-validate every numeric field (rates in [0,1], non-negative counts, cycleP50Ms <= cycleP95Ms) as its own "out_of_range" rejection, distinct from a wrong-typed "malformed" field. - Reject a windowDays that doesn't match the local instance's own resolved window, and a generatedAt outside a bounded freshness window (7 days) or more than 5 minutes in the future. - Enforce decided >= MIN_DECIDED receiver-side rather than trusting the sender's own eligibility claim. - Dedup a batch to the LAST bundle per instanceId (importPeerBundles), then add a persisted, cross-tick per-instance replay/rollback watermark and a per-key Sybil cap (MAX_INSTANCES_PER_KEY) via the new applyFederatedPeerWatermarks, backed by a system_flags JSON blob rather than a new table -- a self-hosted federation's peer count is expected to stay small enough that an in-memory scan/cap-check is trivially cheap. - Move the peer pull off the dashboard's request path entirely: a new "federated-peer-sync" queue job (10-minute cadence, gated on the loopover self-repo's federatedIntelligence.enabled) now runs the pull + trust-gate + persist pipeline AND pushes this instance's own bundle (wiring pushFederatedBundle into a real tick, closing the other half of #9166). buildFederatedBenchmark still computes the local half live (a fast local DB query) but reads the peer half from a cache the tick refreshes. - peerCount now naturally counts distinct contributing INSTANCES (a side effect of the per-instance dedup), matching what its own doc comment always claimed but the old code never enforced.

…r broker enrollments (#9149) issueOrbEnrollment was a bare INSERT: it never revoked, updated, or even counted sibling rows for the same installation_id, so re-running the install flow after a secret leak minted a SECOND simultaneously-valid secret and left the leaked one working forever. The only revoke path sat behind INTERNAL_JOB_TOKEN, unreachable by a self-hosting maintainer. - issueOrbEnrollment takes an optional { rotate: true }: when set, every prior live enrollment for the installation is revoked before the new one is minted. Defaults to false (append) so a blue/green container swap can still rely on two live enrollments briefly overlapping (#9150's sibling fix is what makes that overlap safe, not this one). - The OAuth landing page's callback now recognizes state=<installationId>: rotate / :revoke (installation_id has no channel back through a bare GitHub login/oauth/authorize bounce other than state) -- re-proving admin-of-installation via the SAME verifyInstallationAdmin gate the enrollment flow already uses, so a maintainer can rotate or revoke without operator involvement. Revoke deliberately skips the suspended/self-enrollment-disabled checks: locking down a leaked secret should never be blocked by unrelated administrative state. - The secret page now surfaces how many enrollments are live for the installation and links to the rotate/revoke actions; its response sets Cache-Control: no-store (it was previously a cacheable GET response rendering a plaintext secret). - The internal installations list and enrollment-issue routes gained the same live-enrollment-count visibility and an optional rotate flag, for operator-issued parity with the OAuth self-service path.

codecov · 2026-07-27T09:40:23Z

❌ 16 Tests Failed:

Tests completed	Failed	Passed	Skipped
22963	16	22947	21

View the top 3 failed test(s) by shortest run time

test/unit/selfhost-pg-retention.test.ts > runRetentionPrune + processJob on the Postgres backend (#977) > processJob prune-retention deletes eligible rows and records a success audit event on Postgres

Stack Traces | 0.00509s run time

AssertionError: expected 3 to be +0 // Object.is equality

- Expected
+ Received

- 0
+ 3

 ❯ test/unit/selfhost-pg-retention.test.ts:119:44

test/unit/selfhost-pg-retention.test.ts > pruneExpiredRecords on the Postgres backend (#977) > deletes across multiple bounded batches and stops at the per-table cap, same as the SQLite path

Stack Traces | 0.0127s run time

AssertionError: expected +0 to be 4 // Object.is equality

- Expected
+ Received

- 4
+ 0

 ❯ test/unit/selfhost-pg-retention.test.ts:76:33

test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > skips the miner-identity lookup entirely when the per-repo signal already justifies downgrading

Stack Traces | 0.0145s run time

AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:520:26

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > never downgrades the locally-computed signal when the enabled bridge does not vouch

Stack Traces | 0.0208s run time

AssertionError: expected "vi.fn()" to be called once, but got 2 times
 ❯ test/unit/reputation-wiring.test.ts:441:25

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > keeps the pre-bridge output unchanged when any activation gate is off

Stack Traces | 0.0216s run time

AssertionError: expected "vi.fn()" to not be called at all, but actually been called 1 times

Received:

  1st vi.fn() call:

    Array [
      "https://api.gittensor.io/miners",
      Object {
        "headers": Object {
          "accept": "application/json",
          "user-agent": "loopover/0.1",
        },
        "signal": AbortSignal {
          Symbol(kEvents): Map {},
          Symbol(events.maxEventTargetListeners): 0,
          Symbol(events.maxEventTargetListenersWarned): false,
          Symbol(kHandlers): Map {},
          Symbol(kAborted): false,
          Symbol(kReason): undefined,
          Symbol(kComposite): false,
          Symbol(kTimeout): true,
        },
      },
    ]


Number of calls: 1

 ❯ test/unit/reputation-wiring.test.ts:419:31

test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > widens to the install-wide signal for a CONFIRMED miner when the per-repo signal alone stays neutral

Stack Traces | 0.025s run time

AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:482:26

test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > REGRESSION (#6801): feature on upgrades a low local signal and changes the end-to-end gate decision

Stack Traces | 0.057s run time

AssertionError: expected "vi.fn()" to be called 2 times, but got 3 times
 ❯ test/unit/reputation-wiring.test.ts:393:25

test/unit/selfhost-metrics.test.ts > DEFAULT_METRIC_META completeness (drift guard, 2026-07 fix) > every literal metric name emitted anywhere in src/ has a registered DEFAULT_METRIC_META entry

Stack Traces | 0.157s run time

AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "loopover_orb_relay_multiple_live_enrollments_total",
+ ]

 ❯ test/unit/selfhost-metrics.test.ts:498:21

test/unit/config-templates.test.ts > config/examples review templates (#1682) > lints loopover.full.yml with zero warnings, including no retired top-level fields

Stack Traces | 0.27s run time

AssertionError: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   "gate.mergeReadiness (\"off\") is set alongside an explicitly-authored mode for gate.linkedIssue, gate.duplicates, gate.slop.mode. The composite only fills in a sub-gate mode left unset -- it never overrides an explicitly-configured one, so those fields stay exactly as authored regardless of gate.mergeReadiness.",
+ ]

 ❯ test/unit/config-templates.test.ts:64:29

test/unit/queue-5.test.ts > queue processors > type label decoupling (#label-decoupling) > REGRESSION (#4528, PR #4494 shape): keeps the propagated labels on the PR's own merge-closed webhook, instead of falling back to the title guess

Stack Traces | 0.335s run time

AssertionError: expected [ 'gittensor:feature' ] to deeply equal [ 'gittensor:feature', …(1) ]

- Expected
+ Received

  [
    "gittensor:feature",
-   "gittensor:priority",
  ]

 ❯ test/unit/queue-5.test.ts:6660:34

test/unit/worker-entry-boundary.test.ts > worker entry boundary > does not reference pixelmatch, pngjs, visual-diff, gifenc, or sharp in worker-reachable source

Stack Traces | 0.523s run time

AssertionError: worker-reachable files must not mention Node-only visual diff/GIF/image deps: .../review/visual/visual-findings.ts: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   ".../review/visual/visual-findings.ts",
+ ]

 ❯ test/unit/worker-entry-boundary.test.ts:82:118

View the full list of 5 ❄️ flaky test(s)

test/contract/engine-parity.test.ts > predicted-gate engine parity (#2286) > 'merge-readiness-composite-preserves-e…' matches the committed golden output