fix(orb): webhook redelivery, reputation cadence, DB retention, and gate severity fidelity by JSONbored · Pull Request #9237 · JSONbored/loopover

JSONbored · 2026-07-27T08:29:36Z

Summary

Four independent operational-hygiene fixes, bundled per repo convention:

orb(webhook): 5,547 deliveries stuck 'queued' are permanently un-redeliverable — 4,900 are check_suite.completed, the auto-merge trigger #9054 — stuck queued/superseded webhook deliveries (5,547 rows, 4,900 of them check_suite.completed — the auto-merge trigger) were permanently un-redeliverable: the dedup guard only ever exempted error rows. A row now escapes suppression once it's sat unprocessed past a 10-minute staleness window, and a migration purges the dead historical backlog.
orb(miner): reputation throttle is a permanent, unrecoverable self-ban — cadenceFactor is computed and read by nothing #9062 — the self-reputation cadence throttle was a permanent, unrecoverable self-ban: cadenceFactor was computed and discarded, and any throttled verdict hard-denied open_pr outright. Since submissions are the only source of new decided outcomes, a miner that hit the throttle band could never dilute its ratio back down.
orb(db): 3.4 GB in one month — no retention target has a usable index, and four caches have no delete path at all #9083 — 3.4 GB/month DB growth with no usable retention index (every batched delete forced a full table scan via the ctid/rowid semi-join) and four caches with a read-side TTL but no delete path at all.
orb(gate): finding severity is decorative — 'warning' findings close PRs and a 'critical' one does nothing #9085 — finding severity was decorative: a warning-labeled finding can be the exact reason the gate one-shot-closes a PR, while a critical-labeled one can have zero gate effect (advisory-only by default). An absent AI-judgment confidence also degraded to 1.0 (maximum certainty) at several gate-side sites, and repo_not_registered/repo_unregistered were a one-character, opposite-consequence near-miss pair.

Fixes and judgment calls

#9054 (src/github/webhook.ts, src/db/repositories.ts, migrations/0192_*.sql)

A queued/superseded row past STALE_QUEUED_WEBHOOK_MS (10 min) is treated like an error row: never suppressed, so a GitHub retry or a manual "Redeliver" click re-enqueues it.
Migration purges the historical backlog (rows stuck > 1 day) since replaying is not useful — the code fix means no future delivery can get stuck this way again.
Deferred (documented in the commit): an active cron sweep that proactively re-triggers GitHub's own redelivery API. webhook_events doesn't persist the raw payload by design (to avoid growing the exact blob-retention problem orb(db): 3.4 GB in one month — no retention target has a usable index, and four caches have no delete path at all #9083 addresses), and the staleness-window fix already makes any future stuck delivery redeliverable well inside GitHub's redelivery window without an active poke.

#9083 (src/db/retention.ts, src/db/schema.ts, migrations/0191_*.sql)

Rewrote the batched delete to range over a real, indexable primary key ordered by the retention column instead of rowid/ctid, and added the matching leading index for every policy table.
Added retention for the four never-pruned caches (grounding_file_content_cache at 2 days — its own read-side TTL is 24h — the other three at 30 days) and three previously-untracked append-only logs (review_audit, decision_records at 180 days for dispute evidence, orb_webhook_events).
Deliberately deferred (documented in the commit): pull_request_files, check_summaries, gate_outcomes, agent_runs, advisories are current-state tables (upserted per natural key, read as live state), not append-only logs — pruning them needs a read-pattern review this PR doesn't do.

#9062 (packages/loopover-engine/src/governor/chokepoint.ts, packages/loopover-miner/lib/governor-{chokepoint,state}.ts)

A non-floored cadenceFactor now scales the per-repo write-rate-limit window instead of denying; the hard deny is reserved for the extreme reason === "floored" case only. The miner-lib wrapper advances its bucket against the same scaled policy the decision was evaluated with, so recorded state never disagrees with the verdict it produced.
governor_reputation_history gains a 14-day half-life decay (applied at both read and increment time), so even a floored ratio ages back below the sample-size floor purely from elapsed calendar time, with zero new submissions required.

#9085 (src/rules/advisory.ts, packages/loopover-engine/src/advisory/gate-advisory.ts, src/github/app.ts, src/queue/processors.ts, src/review/{parity-wire,unified-comment-bridge}.ts)

formatCheckRunOutput/buildCheckRunAnnotations now accept the real blocker codes from the same advisory's GateCheckEvaluation (optional, purely additive — every existing caller that omits it keeps byte-identical rendering). When provided, an actual blocker always renders at the most alarming level regardless of its authored severity, and a non-blocking finding is capped at warning even if authored critical. Wired the one caller that already has the evaluation in scope (the check-run publish in processors.ts).
- Judgment call: I did not attempt to make severity load-bearing in the actual merge/close decision (evaluateGateCheckCore) — that logic is already correctly driven by isConfiguredGateBlocker/policy modes, completely independent of the severity field, and rewiring severity itself to drive blocking would be a much larger, riskier behavioral change to a production auto-merge/close path than this PR should attempt. The issue's own complaint — "a contributor reading a warning icon on the finding that is about to close their PR is being actively misinformed" — is specifically about the displayed icon not matching the real consequence, which this fixes without touching the decision path at all.
- Taxonomy registration for slop_risk_above_threshold/surface_lane_reject (item 2 of the issue) was already fixed on main by fix(ai-review): verify reviewers actually agree; strip blocker authority from a bailed review #9114 prior to this branch.
An absent AdvisoryFinding.confidence now degrades to CONFIDENCE_WHEN_UNSTATED (0.5) instead of 1.0 at the three gate-side consumption sites in advisory.ts (and the mirrored site in the engine twin) — matching the "silence is not certainty" fix CONFIDENCE_WHEN_UNSTATED already applied at the model-parsing layer.
Renamed repo_not_registered → repo_not_cached (mirrors pr_not_cached/issue_not_cached's naming for the identical "not yet synced" shape) to resolve the near-miss with repo_unregistered, which fires when a repo record exists but isn't in the Gittensor registry snapshot — a plain non-blocking advisory warning, the opposite of repo_not_cached's app-state hold. Renamed consistently across both src/rules/advisory.ts and the engine twin (gate-advisory.ts), plus their consumers (parity-wire.ts, unified-comment-bridge.ts) and tests.

Validation

npx tsc --noEmit clean on the host, @loopover/engine, and @loopover/miner packages.
npm run build --workspace @loopover/engine clean.
npx tsx scripts/check-engine-parity.ts — ok (7 hand-duplicated pairs agree).
Full @loopover/engine test suite (npm run test --workspace @loopover/engine): 739/739 pass.
npx vitest run test/contract/engine-parity.test.ts: 14/14 pass.
Directly relevant vitest files (rules, unified-comment-bridge, parity-wire, predicted-gate-engine, slop, scenario-summary, webhook, retention, miner-attempt-input-builder, miner-governor-chokepoint, miner-governor-state, github-app): 622/622 pass, including new regression tests for each fix.
Did not run the full npm run test:coverage/test:ci gate per the scope of this pass — happy to run it on request.

Closes #9054
Closes #9062
Closes #9083
Closes #9085

…pruned caches (#9083) No RETENTION_POLICY table had a leading index on its retention timestamp, so pruneExpiredRecords's ctid-keyed batched delete forced a full sequential scan of the whole table on Postgres per batch -- large tables blew the prune job's 30-minute timeout and retention fell permanently behind (signal_snapshots alone reached 787MB at ~44KB/row). Rewrite the delete to range over a real, indexable primary key ordered by the retention column instead of rowid/ctid, and add the matching leading index for every policy table via migration 0191. Also close the "no delete path at all" gap for four read-side caches (grounding_file_content_cache, ai_review_cache, ai_slop_cache, linked_issue_satisfaction_cache) that only enforced their TTL at the read site, and add retention for three previously-untracked append-only logs (review_audit, decision_records, orb_webhook_events). Deliberately deferred: pull_request_files, check_summaries, gate_outcomes, agent_runs, and advisories are current-state tables (upserted per natural key, read as live state for open PRs/checks) rather than append-only logs, so they are excluded from this pass per RETENTION_POLICY's own documented scope -- pruning them needs a read-pattern review this PR doesn't do.

…manent duplicates (#9054) The dedup guard in enqueueWebhookByEnv only ever exempted 'error' rows from suppression. A row left at 'queued' (the insert happened but WEBHOOKS.send() was lost) or 'superseded' (overwritten by a later coalesced delivery) could never be redelivered: every GitHub retry and every operator "Redeliver" click carries the same delivery_id + payload hash, hits the guard, and is silently discarded as a no-op duplicate forever -- including check_suite.completed, the maybeReReviewOnCiCompletion auto-merge trigger. getWebhookEvent now also returns receivedAt, and a 'queued'/'superseded' row past a 10-minute staleness window is treated the same as an 'error' row: never suppressed. A migration purges the historical backlog of rows this bug already stuck permanently (dead by definition once a full day old), since replaying them is not useful and the code fix means no future delivery can get stuck this way again. Deferred: an active cron sweep that proactively re-triggers GitHub's own redelivery API for still-stuck rows was not implemented -- webhook_events does not persist the raw payload (by design, to avoid growing the exact blob-retention problem #9083 addresses), and this fix already ensures any future stuck delivery is redeliverable well inside GitHub's redelivery window without needing an active poke.

…ead of hard-denying (#9062) selfReputationThrottle's own doc comment promises a soft cadence throttle that "a recovering ratio restores it -- never a hard permanent ban", but the chokepoint ignored cadenceFactor entirely and hard-denied on ANY throttled verdict. Since submissions are the only source of new decided outcomes, and governor_reputation_history never decayed, a miner that hit the throttle band could never submit again to dilute its ratio back down -- an absorbing, permanent self-ban. evaluateGovernorChokepoint now consumes a non-floored cadenceFactor by scaling the per-repo write-rate-limit window instead of denying; the hard deny is reserved for the extreme "floored" ratio only. The miner-lib wrapper advances its rate-limit bucket against the SAME scaled policy the decision was evaluated with, so state stays consistent with the verdict it recorded. governor_reputation_history also gains a 14-day half-life decay (applied at both read and increment time), so even a floored ratio ages back below the sample-size floor over calendar time with zero new submissions, delivering the "recovering ratio" contract the module already documented.

… tighten confidence default, resolve near-miss code names (#9085) Finding severity was purely decorative: a warning-labeled finding (missing_linked_issue, slop_risk_above_threshold) can be the exact reason the gate one-shot-closes a PR, while a critical-labeled one (ai_consensus_defect, under the advisory-only default aiReviewGateMode) can have no gate effect at all. A contributor reading a plain warning icon on the finding that is about to close their PR was being actively misinformed. formatCheckRunOutput/buildCheckRunAnnotations now accept the real blocker codes from the SAME advisory's GateCheckEvaluation (when the caller has one) and use them to correct the displayed severity: an actual blocker always renders at the most alarming level regardless of its authored severity, and a non-blocking finding is capped at warning even if authored critical, so it never cries wolf louder than a genuine one. The parameter is optional and purely additive -- every existing caller that omits it keeps today's exact rendering. Wired the one caller with the evaluation already in scope (processors.ts's check-run publish). Also: - An absent AdvisoryFinding.confidence degraded to 1.0 (maximum certainty) at three gate-side consumption sites, the same "silence is not certainty" anti-pattern CONFIDENCE_WHEN_UNSTATED already fixed at the model-parsing layer. These sites read confidence off a finding object via a path the parser doesn't cover (a producer that omitted it, or the unvalidated cached-advisory JSON parse), so they now share the same 0.5 fallback instead of a second, wrong hardcoded default. - Renamed the near-miss repo_not_registered -> repo_not_cached (mirrors pr_not_cached/issue_not_cached's naming for the identical "not yet synced" shape) to stop it being confused with repo_unregistered, one character away and with the opposite gate consequence (repo_not_cached holds the gate for a human; repo_unregistered is a non-blocking advisory warning). The engine's gate-advisory.ts twin gets the matching confidence-default and rename fixes to stay in lock-step with the host copy.

github-actions · 2026-07-27T08:30:53Z

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (03b6418) and head (eae7358) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: `linked_issue_scope_mismatch`

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

cloudflare-workers-and-pages · 2026-07-27T08:31:09Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	`eae7358`	Commit Preview URL Branch Preview URL	Jul 27 2026, 08:31 AM

codecov · 2026-07-27T08:31:28Z

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

loopover-orb · 2026-07-27T08:35:08Z

Caution

🛑 LoopOver review result - fixes required

_{Review updated: 2026-07-27 09:07:24 UTC}

26 files · 1 AI reviewer · 1 blocker · CI failing · unstable

🛑 Suggested Action - Manual Review

Review summary
The AI review returned blocking findings for this change but did not include a separate narrative summary. Review the blockers below before deciding this PR.

Blockers

src/db/schema.ts (webhookEvents table) adds `status: index("webhook_events_status_idx").on(table.status)` as a brand-new index (the table previously had no indexes block at all), but migrations/0191_retention_column_indexes.sql only issues `CREATE INDEX IF NOT EXISTS idx_webhook_events_received_at ON webhook_events(received_at);` — the status index is declared in the drizzle schema with no matching `CREATE INDEX` anywhere in this PR's migrations, so it will never actually exist on D1/SQLite until a follow-up migration adds it.

Nits — 6 non-blocking

migrations/0192_purge_stuck_queued_webhook_events.sql compares `received_at < datetime('now', '-1 day')` against a column populated by `nowIso()` (ISO-8601 with a `T` separator) — `datetime('now', ...)` emits a space-separated SQLite datetime string, so on the boundary day the lexicographic comparison can misclassify a same-day row (per this repo's own documented ISO-vs-datetime comparison pitfall); low impact here since it's a one-time cleanup sweep, but worth using an ISO cutoff literal instead.
packages/loopover-engine/src/governor/chokepoint.ts:134 introduces the `0.001` divide-by-zero floor as an inline literal — consider hoisting it to a named constant alongside `REPUTATION_HISTORY_HALF_LIFE_DAYS`-style constants for discoverability.
src/db/retention.ts's new retention windows (2/30/90/180 days) are well-commented inline but are raw numeric literals repeated across several RETENTION_POLICY entries — a small `DAYS` helper or named constants would make future edits less error-prone, though this matches the file's pre-existing style.
Worth double-checking that no UI/OpenAPI surface, docs, or config example still references the old `repo_not_registered` finding code outside the files shown in this diff, since the rename touches a public-ish advisory code.
Add `CREATE INDEX IF NOT EXISTS webhook_events_status_idx ON webhook_events(status);` to migrations/0191_retention_column_indexes.sql (or a new 0193 migration) to close the schema/migration gap for the new status index.
PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Why this is blocked

src/db/schema.ts (webhookEvents table) adds `status: index("webhook_events_status_idx").on(table.status)` as a brand-new index (the table previously had no indexes block at all), but migrations/0191_retention_column_indexes.sql only issues `CREATE INDEX IF NOT EXISTS idx_webhook_events_received_at ON webhook_events(received_at);` — the status index is declared in the drizzle schema with no matching `CREATE INDEX` anywhere in this PR's migrations, so it will never actually exist on D1/SQLite until a follow-up migration adds it.

📋 Copy for AI agents — paste into your coding agent

Fix the following blocker(s) from this PR review:

1. src/db/schema.ts \(webhookEvents table\) adds \`status: index\("webhook\_events\_status\_idx"\).on\(table.status\)\` as a brand-new index \(the table previously had no indexes block at all\), but migrations/0191\_retention\_column\_indexes.sql only issues \`CREATE INDEX IF NOT EXISTS idx\_webhook\_events\_received\_at ON webhook\_events\(received\_at\);\` — the status index is declared in the drizzle schema with no matching \`CREATE INDEX\` anywhere in this PR's migrations, so it will never actually exist on D1/SQLite until a follow-up migration adds it.

CI checks failing

validate
validate-tests
validate-code

Decision drivers

❌ Code review — 1 blocker (1 reviewer)
❌ Gate result — Blocking (Repo-configured hard blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#9054, #9062, #9083, #9085
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	❌ 8/20	High review scope from cached public metadata (4 linked issues).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 13 registered-repo PR(s), 13 merged, 331 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 13 PR(s), 331 issue(s).
Improvement	✅ Minor	risk: clean · value: minor

Linked issue satisfaction

Addressed
The diff directly implements the issue's core ask in src/github/webhook.ts: a queued/superseded row past a 10-minute staleness window now bypasses the dedup guard (isStaleStuck), and migration 0192 purges the historical stuck backlog per the issue's own suggested one-off. The acceptance criterion (kill process between insert and send, then redeliver) is satisfied by this exact staleness check, tho

Review context

Author: JSONbored
Role context: owner (maintainer lane)
Public audience mode: oss maintainer
Lane context: Repository is configured for direct PR review.
Public profile languages: Python, TypeScript, Ruby, Go, MDX, Shell, Solidity, JavaScript
Official Gittensor activity: 13 PR(s), 331 issue(s).
PR-specific overlap: none found.

Contributor next steps

Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.
Then work through the remaining 2 steps in the Signals table above.

Signal definitions

Related work = same linked issue, overlapping active PRs, or title/path similarity.
Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

@loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
@loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Decision record

action: hold · clause: ai_consensus_defect
config: 03a7f8b529a9 · pack: oss-anti-slop
model: n/a · prompt: 32528cb0d9b8 · confidence: 0.75
record: 2263b5985ab0 (schema v3, head eae7358)

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

Re-run LoopOver review

JSONbored · 2026-07-27T08:54:56Z

This finding is a false positive: `webhook_events_status_idx` isn't new to this PR — it was created back in `migrations/0001_initial.sql:143` (`CREATE INDEX IF NOT EXISTS webhook_events_status_idx ON webhook_events (status);`). This PR only adds the `received` index (`idx_webhook_events_received_at`, migration 0191), which the comment right above it in `schema.ts` says explicitly ("only the status index above existed before"). Drizzle requires declaring all of a table's indexes in one block, so the reviewer saw both listed together in the same schema.ts hunk and understandably assumed both were new. `npm run db:schema-drift:check` confirms schema.ts and migrations/ already agree with no drift.

#9235 and #9237 merged 36 seconds apart, each independently claiming migration number 0191 (0191_linked_issue_claims.sql and 0191_retention_column_indexes.sql). Renumbers the latter (the one that merged second) to the next free number, 0193.

JSONbored added 4 commits July 27, 2026 01:25

JSONbored self-assigned this Jul 27, 2026

loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 27, 2026

loopover-orb Bot added the manual-review Gittensor contributor context label Jul 27, 2026

JSONbored merged commit d2c8d27 into main Jul 27, 2026
5 of 8 checks passed

JSONbored deleted the fix/9054-9062-9083-9085-operational-hygiene branch July 27, 2026 09:07

JSONbored mentioned this pull request Jul 27, 2026

fix(db): renumber the duplicate 0191 migration to 0193 #9248

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(orb): webhook redelivery, reputation cadence, DB retention, and gate severity fidelity#9237

fix(orb): webhook redelivery, reputation cadence, DB retention, and gate severity fidelity#9237
JSONbored merged 4 commits into
mainfrom
fix/9054-9062-9083-9085-operational-hygiene

JSONbored commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 27, 2026

Uh oh!

codecov Bot commented Jul 27, 2026

Uh oh!

loopover-orb Bot commented Jul 27, 2026 •

edited

Loading

🛑 LoopOver review result - fixes required

Uh oh!

JSONbored commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JSONbored commented Jul 27, 2026

Summary

Fixes and judgment calls

Validation

Uh oh!

github-actions Bot commented Jul 27, 2026

Logic backtest

Backtest comparison: linked_issue_scope_mismatch

Uh oh!

cloudflare-workers-and-pages Bot commented Jul 27, 2026

Deploying with Cloudflare Workers

Uh oh!

codecov Bot commented Jul 27, 2026

⚠️ JUnit XML file not found

Uh oh!

loopover-orb Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 LoopOver review result - fixes required

Uh oh!

JSONbored commented Jul 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Backtest comparison: `linked_issue_scope_mismatch`

loopover-orb Bot commented Jul 27, 2026 •

edited

Loading