Skip to content

fix(orb): repoint the orphaned review_targets reputation signal, sample clock skew in broker mode, and bound the upstream-drift fallback fetch#9230

Merged
JSONbored merged 4 commits into
mainfrom
fix/9136-9156-9165-observability-sentinels
Jul 27, 2026
Merged

fix(orb): repoint the orphaned review_targets reputation signal, sample clock skew in broker mode, and bound the upstream-drift fallback fetch#9230
JSONbored merged 4 commits into
mainfrom
fix/9136-9156-9165-observability-sentinels

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Bundles three related "a placeholder silently satisfies its own health check" observability fixes.

#9165 — raw-GitHub fallback fetch had no timeout

fetchTrackedSource's raw-GitHub fallback (src/upstream/ruleset.ts:520) was the only fetch in the
file without a timeout; a stalled raw.githubusercontent.com connection hung the whole scheduled
refreshUpstreamDrift job. It now goes through the same timeoutFetch every other call site in the
file already uses. Confirmed the Promise.all fan-out in refreshUpstreamSourceSnapshots degrades
per-source (a test simulates one source timing out while five others complete normally).

Swept src/ for the same bare-fetch shape and found four more real gaps, all fixed the same way:

  • src/auth/github-oauth.ts — device-flow start/poll + web-OAuth code exchange (3 call sites)
  • src/integrations/linear-adapter.ts — the Linear GraphQL call
  • src/registry/sync.ts — the registry-probe candidate loop

Everything else grep-matched already carried a signal: AbortSignal.timeout(...) (verified site by
site — qdrant-vectorize.ts, ai.ts, capture.ts, actions-fallback.ts, notify-*.ts, gittensor/api.ts, etc.).

#9156 — clock skew never sampled in broker mode, staleness gauge wired to nothing

recordClockSkewFromResponse's only call site was inside the local GitHub App JWT mint path
(src/github/app.ts:224), which a brokered self-host can never reach — mintInstallationToken
returns from the broker branch first, so a brokered deployment's clock-skew gauge read a hard 0,
indistinguishable from "clock is fine". Now sampled from fetchBrokeredInstallationToken's own
response too (both the success and failure arm — the Date header is there either way).

0 is a real, reachable skew value (a perfectly synced clock), so it can't double as "unmeasured" —
clockSkewSecondsSample() now returns NaN until the first real sample lands.

The companion loopover_clock_skew_sample_age_seconds gauge (added in #7000 precisely so a stale
reading is distinguishable from a fresh one) was referenced by zero alert rules and zero dashboard
panels. Added LoopoverClockSkewSampleStale (fires past 2h — a ~2x margin over the roughly-hourly
token-mint cadence) plus the matching Grafana stat panel. Audited the sibling -1-sentinel gauge
(loopover_orb_relay_drain_seconds_since_last, #9128) — it already has an alert rule from a prior PR,
so nothing further needed there.

#9136 — orphaned review_targets (largest of the three)

review_targets has had no live writer since the 2026-06-22 convergence cutover. A prior PR (#9179)
already repointed the anomaly-alerter's reversal and DLQ signals in ops.ts onto the live
review_audit ledger (and fixed the target_id namespace mismatch in that join), landing
checkReviewSourceFreshness as the generalizable staleness check. That PR explicitly deferred the
rest of the scope to a follow-up — this PR is that follow-up.

Decision: repoint, not restore a writer — same call #9179 already made for the reversal/DLQ half;
this PR applies it consistently rather than re-litigating.

Full reader inventory (per the issue's request to enumerate every one):

Reader Status
ops.ts reversals + DLQ signals (computeAgentHealth) repointed in #9179 (prior PR)
ops.ts checkReviewSourceFreshness added in #9179 (prior PR)
submitter-reputation.tsgetSubmitterReputation repointed here
submitter-reputation.tsgetSubmitterReputationAcrossInstall repointed here
submitter-reputation.tslistSubmitterCohortRows repointed here
ams-miner-cohort.ts no code change needed — consumes listSubmitterCohortRows directly, picks up the repoint for free; stale doc comments updated
evaluateVisualVisionGate / evaluateScreenshotTableVisionGate low_reputation skip given a compensating advisory finding here (see below)
ops.ts computeAgentHealthbyStatus/byVerdict/failedRows/manualRate/stuckRetryable/failed still review_targets-sourced — deferred, see below
ops.ts computeCalibration — merged/closes-by-reason/disputed still review_targets-sourced — deferred, see below
public-stats.ts already live-ledger-sourced (predates this issue)

What's repointed: getSubmitterReputation, getSubmitterReputationAcrossInstall, and
listSubmitterCohortRows moved off review_targets's 90-day terminal_at window (which was heading
to permanently-empty around 2026-09-20) onto review_audit's pr_outcome rows joined to
pull_requests for the submitter's login — the same live ledger getSubmitterCadence (#9015) and
#9179's reversal/DLQ signals already read. Restricted to each target's latest pr_outcome row, since
recordPrOutcome's webhook path has no existence check (unlike the direct-write path) and a
redelivered closed webhook can double-insert.

avgAttemptCount in listSubmitterCohortRows is a documented substitution, not a silent behavior
change: the pre-repoint review_targets.attempt_count (the gate's own re-review counter) has no live
equivalent, so it now reads pull_requests.merge_attempt_count — narrower (only failed merge attempts:
permission/check/conflict, reset per head SHA) but still a genuine review-friction signal. Called out
in both submitter-reputation.ts and ams-miner-cohort.ts's doc comments.

Compensating hold: restoring the reputation signal without addressing the two vision-gate skips
would reproduce #9015's "suspicion buys less scrutiny" shape — a low-reputation submitter's low_reputation
skip in evaluateVisualVisionGate / evaluateScreenshotTableVisionGate previously vanished with no trace.
Both now push a severity: "warning" advisory finding (never a blocker — both modules stay strictly
advisory) naming how many real routes/pairs were skipped, wired into processors.ts's
runVisualVisionForAdvisory / runScreenshotTableVisionForAdvisory. No finding is pushed when there was
nothing to compensate for (no confirmed regression / no real image pairs) — that's a legitimately
different skip reason, not reputation-driven.

Deferred, still on review_targets: computeAgentHealth's byStatus/byVerdict/failedRows/
manualRate/stuckRetryable/failed and computeCalibration's merged/closes-by-reason/disputed
queries. These aren't a mechanical repoint like the pieces above — review_targets' non-terminal
states (queued/reviewing/error/error_retryable) and its attempt-exhausted failed bucket have
no live per-target equivalent post-cutover: gate_decision's own decision column
(src/review/parity.ts's GateAction) only ever records 'merge' | 'close' | 'hold', not the full
status enum review_targets tracked. Repointing these needs a product decision on what a manual/
error/stuck bucket even means without a live gate-processing-state ledger, which I didn't want to
guess at silently in the same PR as the mechanical repoints above. checkReviewSourceFreshness's
review_targets comment is updated to point at these as the actual remaining live readers (it
previously, now-incorrectly, cited submitter-reputation.ts).

Practical effect: the anomaly alerter's DLQ/reversal signals (fixed in #9179) and the reputation
quality signal + AMS-cohort dashboard (fixed here) are live again. manualRate/stuckRetryable/
failed in the /status alerter remain silently zero until the deferred piece above is resolved —
called out explicitly rather than left to be rediscovered.

Testing

  • npx tsc --noEmit -p tsconfig.json --incremental false — clean
  • Targeted vitest run across every touched/consuming test file (clock-skew, orb-broker-client,
    upstream-ruleset, submitter-reputation, visual-findings, screenshot-table-vision(-wiring)x2,
    visual-vision-wiring, ops, ams-miner-cohort, mcp-cli-ams-miner-cohort, registry, linear-adapter,
    auth, auth-github-token, alerts-metric-name-references, observability-ci,
    selfhost-grafana-dashboard, validate-observability-configs-script,
    docs-selfhost-troubleshooting-metric-names) — 549 tests, all green
  • Rebased cleanly onto origin/main (4 commits, no overlap with touched files)

Closes #9165
Closes #9156
Advances #9136 (see the deferred-scope note above — ops.ts's remaining review_targets readers are
intentionally out of scope for this PR)

@JSONbored JSONbored self-assigned this Jul 27, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 27, 2026

Copy link
Copy Markdown

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
22306 6 22300 21
View the top 3 failed test(s) by shortest run time
test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > skips the miner-identity lookup entirely when the per-repo signal already justifies downgrading
Stack Traces | 0.0179s run time
AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:520:26
test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > keeps the pre-bridge output unchanged when any activation gate is off
Stack Traces | 0.0196s run time
AssertionError: expected "vi.fn()" to not be called at all, but actually been called 1 times

Received:

  1st vi.fn() call:

    Array [
      "https://api.gittensor.io/miners",
      Object {
        "headers": Object {
          "accept": "application/json",
          "user-agent": "loopover/0.1",
        },
        "signal": AbortSignal {
          Symbol(kEvents): Map {},
          Symbol(events.maxEventTargetListeners): 0,
          Symbol(events.maxEventTargetListenersWarned): false,
          Symbol(kHandlers): Map {},
          Symbol(kAborted): false,
          Symbol(kReason): undefined,
          Symbol(kComposite): false,
          Symbol(kTimeout): true,
        },
      },
    ]


Number of calls: 1

 ❯ test/unit/reputation-wiring.test.ts:419:31
test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > never downgrades the locally-computed signal when the enabled bridge does not vouch
Stack Traces | 0.0198s run time
AssertionError: expected "vi.fn()" to be called once, but got 2 times
 ❯ test/unit/reputation-wiring.test.ts:441:25
test/unit/reputation-wiring.test.ts > getEffectiveSubmitterReputation (#4513, install-wide for a confirmed miner) > widens to the install-wide signal for a CONFIRMED miner when the per-repo signal alone stays neutral
Stack Traces | 0.03s run time
AssertionError: expected 'neutral' to be 'low' // Object.is equality

Expected: "low"
Received: "neutral"

 ❯ test/unit/reputation-wiring.test.ts:482:26
test/unit/reputation-wiring.test.ts > ORB/AMS reputation bridge wiring (#6801) > REGRESSION (#6801): feature on upgrades a low local signal and changes the end-to-end gate decision
Stack Traces | 0.0413s run time
AssertionError: expected "vi.fn()" to be called 2 times, but got 3 times
 ❯ test/unit/reputation-wiring.test.ts:393:25
test/unit/worker-entry-boundary.test.ts > worker entry boundary > does not reference pixelmatch, pngjs, visual-diff, gifenc, or sharp in worker-reachable source
Stack Traces | 0.606s run time
AssertionError: worker-reachable files must not mention Node-only visual diff/GIF/image deps: .../review/visual/visual-findings.ts: expected [ Array(1) ] to deeply equal []

- Expected
+ Received

- []
+ [
+   ".../review/visual/visual-findings.ts",
+ ]

 ❯ test/unit/worker-entry-boundary.test.ts:82:118

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 27, 2026
@loopover-orb

loopover-orb Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Caution

🛑 LoopOver review result - fixes required

Review updated: 2026-07-27 08:56:08 UTC

23 files · 1 AI reviewer · no blockers · CI failing · unstable

🛑 Suggested Action - Manual Review

Review summary
This PR bundles three distinct observability fixes: repointing the reputation signal off the orphaned review_targets table onto live review_audit/pull_requests ledgers, sampling clock skew in broker mode (with a 0→NaN sentinel fix), and bounding the raw-GitHub fallback fetch plus a sweep of four other unbounded fetch call sites. Each fix traces cleanly to a real, previously-unreachable/broken path (review_targets frozen since 2026-06-22 cutover, recordClockSkewFromResponse unreachable in broker mode, bare fetch with no timeout), and each has direct unit tests exercising the actual new code path rather than fabricated scenarios (e.g. the LATEST pr_outcome dedup test, the broker success/failure Date-header sampling test, the partial-timeout Promise.all degradation test). The PR_OUTCOME_JOIN's string-concatenation-based join (`po.project || '#' || pr.number = po.target_id`) is a reasonable documented substitution given review_audit's lack of an author column, and the avgAttemptCount substitution (merge_attempt_count for attempt_count) is explicitly and honestly documented as narrower rather than silently changing meaning. The compensating advisory findings for low-reputation vision skips are new-but-narrow, well-scoped, and tested for both the confirmed-regression and no-regression branches.

Nits — 6 non-blocking

CI checks failing

  • validate
  • validate-tests

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #9165, #9156
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (2 linked issues).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 13 registered-repo PR(s), 13 merged, 331 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 13 PR(s), 331 issue(s).
Improvement ✅ Minor risk: clean · value: minor
Linked issue satisfaction

Addressed
The diff switches ruleset.ts:520's raw-GitHub fallback fetch to timeoutFetch, adds a comment/test confirming the Promise.all fan-out degrades per-source rather than hanging, and sweeps and fixes four other bare-fetch call sites across the codebase (github-oauth.ts, linear-adapter.ts, registry/sync.ts), matching the issue's core requirement and sweep ask.

Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
  • Official Gittensor activity: 13 PR(s), 331 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record
  • action: hold · clause: success
  • config: 03a7f8b529a9 · pack: oss-anti-slop
  • record: bae68c608f64 (schema v3, head a63a1d2)

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 27, 2026
@JSONbored
JSONbored force-pushed the fix/9136-9156-9165-observability-sentinels branch from a63a1d2 to bf4a216 Compare July 27, 2026 08:56
)

fetchTrackedSource's raw-GitHub fallback was the only fetch in ruleset.ts
without a timeout, so a stalled raw.githubusercontent.com connection hung
the whole scheduled refreshUpstreamDrift job. Route it through the same
timeoutFetch every other call site in the file already uses, and confirm
the Promise.all fan-out degrades per-source rather than all-or-nothing.

Sweep the rest of src/ for the same bare-fetch shape: three GitHub calls
in github-oauth.ts (device-flow start/poll, web-OAuth code exchange) and
one call each in linear-adapter.ts and registry/sync.ts had no timeout
and are now bounded the same way.
#9156)

recordClockSkewFromResponse's only call site is inside the local GitHub
App JWT mint path, which a brokered self-host never reaches (mintInstallationToken
returns from the broker branch first) -- so a brokered deployment's clock
skew gauge read a hard 0, indistinguishable from "clock is fine". Sample
it from fetchBrokeredInstallationToken's own response instead, on both
success and failure.

0 is a real, reachable skew value (a perfectly synced clock), so it can't
double as "never sampled" -- clockSkewSecondsSample() now returns NaN
until the first real sample lands.

The companion loopover_clock_skew_sample_age_seconds gauge (added in #7000
precisely so a stale reading is distinguishable from a fresh one) was wired
to zero alert rules and zero dashboard panels. Add LoopoverClockSkewSampleStale
(fires past 2h, tolerating normal ~hourly mint-timing jitter) and the
matching Grafana stat panel.
…w_targets (#9136)

getSubmitterReputation, getSubmitterReputationAcrossInstall, and
listSubmitterCohortRows read `review_targets` over a 90-day recency
window -- but review_targets has had no live writer since the 2026-06-22
convergence cutover, so that window reads a shrinking set that goes
permanently empty around 2026-09-20 (following #9015's partial fix and
#9179's repoint of the anomaly-alerter's reversal/DLQ signals).

Repointed onto the same live ledgers #9179 and getSubmitterCadence (#9015)
already read: review_audit's pr_outcome rows (the realized outcome, not
just the bot's own prediction) joined to pull_requests for the submitter's
login, restricted to each target's LATEST pr_outcome row so a redelivered
webhook can't double-count. ams-miner-cohort.ts required no changes --
it consumes listSubmitterCohortRows directly and picks the repoint up
for free; its stale doc comments are updated to match.

Before restoring the signal, give evaluateVisualVisionGate's and
evaluateScreenshotTableVisionGate's `low_reputation` skip a compensating
advisory finding: silently skipping the one extra AI check a low-reputation
submitter's confirmed visual change would otherwise get reproduces #9015's
"suspicion buys less scrutiny" shape.

Remaining scope, tracked under #9136: ops.ts's computeAgentHealth
(byStatus/byVerdict/failedRows/manualRate/stuckRetryable/failed) and
computeCalibration still read review_targets directly. Deferred because
review_targets' non-terminal states (queued/reviewing/error/error_retryable)
and the attempt-exhausted 'failed' bucket have no live per-target
equivalent post-cutover -- gate_decision's own `decision` column only
ever records 'merge' | 'close' | 'hold', not the full status enum
review_targets tracked. See the PR description for the full reader
inventory.
@JSONbored
JSONbored force-pushed the fix/9136-9156-9165-observability-sentinels branch from bf4a216 to f3d151a Compare July 27, 2026 09:00
@JSONbored
JSONbored merged commit ad8975e into main Jul 27, 2026
3 of 4 checks passed
@JSONbored
JSONbored deleted the fix/9136-9156-9165-observability-sentinels branch July 27, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

1 participant