Summary
The hosted D1 (gittensory, b2c79dd6-…) reached its 10 GB hard maximum. Every write then failed with D1_ERROR: Exceeded maximum DB size, including recordOrbWebhookEvent's INSERT — so /v1/orb/webhook returned HTTP 500 to GitHub and inbound webhook delivery stopped fleet-wide for ~4.5 hours (12:10 → 16:56 UTC on 2026-07-27).
Per the maintainer this has recurred roughly every 1–2 weeks.
Impact while down
No webhooks reached the self-host: the re-run checkbox did nothing, PR events were not ingested, and reviews only continued because the internal regate sweep does not depend on webhooks. GitHub's delivery log showed every attempt returning 500 {"error":"event_record_failed"}.
Why it recurs (structural, three compounding causes)
- Five high-growth tables had no retention rule at all. Measured during the incident:
check_summaries 100,459 rows / 0.30GB payload_json; pull_request_files 54,532 / 0.20GB; repo_github_totals_snapshots 198,552; recent_merged_pull_requests 29,634; orb_pr_outcomes 9,148.
MAX_DELETED_PER_TABLE = 50_000 against ~4.3M rows written/day (measured via wrangler d1 info). A daily prune could delete at most ~1.25M rows/day across the whole policy — a permanent ~3.5× deficit in which the cap, not the cutoff, was the binding constraint. No window tightening alone could close it.prune-retention ran once a day (03:00 UTC).
Immediate mitigation (manual, already applied)
Pruned ~908 MB of aged rows (9.99 GB → 9.09 GB), oldest-first, from check_summaries, pull_request_files, repo_github_totals_snapshots, orb_webhook_events, webhook_events. Webhooks resumed immediately; orb_relay_pending drains to 0.
Useful finding: D1 does reclaim space on DELETE (size_after drops per batch), so an export/recreate migration was not required.
Permanent fix
PR #9415 — adds the five tables to RETENTION_POLICY, raises the cap to 250k, moves prune to hourly, and tightens both webhook idempotency logs 90d → 14d.
Remaining gap (why this issue stays open)
There is no alerting on D1 size. This class of failure surfaces as a silent fleet-wide outage discovered by a human noticing reviews stopped. Even with #9415 merged, nothing warns before the ceiling is hit again.
Proposed: alert at ~70% and ~85% of the 10 GB cap. wrangler d1 info exposes database_size, and there is an existing D1 size probe (src/selfhost/d1-size-probe.ts, gated on CLOUDFLARE_D1_MONITOR_*) that appears to be unconfigured — worth wiring rather than building new.
Note PRAGMA page_count / freelist_count are blocked by D1 (SQLITE_AUTH), so size must come from the Cloudflare API, not from SQL.
Summary
The hosted D1 (
gittensory,b2c79dd6-…) reached its 10 GB hard maximum. Every write then failed withD1_ERROR: Exceeded maximum DB size, includingrecordOrbWebhookEvent's INSERT — so/v1/orb/webhookreturned HTTP 500 to GitHub and inbound webhook delivery stopped fleet-wide for ~4.5 hours (12:10 → 16:56 UTC on 2026-07-27).Per the maintainer this has recurred roughly every 1–2 weeks.
Impact while down
No webhooks reached the self-host: the re-run checkbox did nothing, PR events were not ingested, and reviews only continued because the internal regate sweep does not depend on webhooks. GitHub's delivery log showed every attempt returning 500
{"error":"event_record_failed"}.Why it recurs (structural, three compounding causes)
check_summaries100,459 rows / 0.30GBpayload_json;pull_request_files54,532 / 0.20GB;repo_github_totals_snapshots198,552;recent_merged_pull_requests29,634;orb_pr_outcomes9,148.MAX_DELETED_PER_TABLE = 50_000against ~4.3M rows written/day (measured viawrangler d1 info). A daily prune could delete at most ~1.25M rows/day across the whole policy — a permanent ~3.5× deficit in which the cap, not the cutoff, was the binding constraint. No window tightening alone could close it.prune-retentionran once a day (03:00 UTC).Immediate mitigation (manual, already applied)
Pruned ~908 MB of aged rows (9.99 GB → 9.09 GB), oldest-first, from
check_summaries,pull_request_files,repo_github_totals_snapshots,orb_webhook_events,webhook_events. Webhooks resumed immediately;orb_relay_pendingdrains to 0.Useful finding: D1 does reclaim space on DELETE (
size_afterdrops per batch), so an export/recreate migration was not required.Permanent fix
PR #9415 — adds the five tables to
RETENTION_POLICY, raises the cap to 250k, moves prune to hourly, and tightens both webhook idempotency logs 90d → 14d.Remaining gap (why this issue stays open)
There is no alerting on D1 size. This class of failure surfaces as a silent fleet-wide outage discovered by a human noticing reviews stopped. Even with #9415 merged, nothing warns before the ceiling is hit again.
Proposed: alert at ~70% and ~85% of the 10 GB cap.
wrangler d1 infoexposesdatabase_size, and there is an existing D1 size probe (src/selfhost/d1-size-probe.ts, gated onCLOUDFLARE_D1_MONITOR_*) that appears to be unconfigured — worth wiring rather than building new.Note
PRAGMA page_count/freelist_countare blocked by D1 (SQLITE_AUTH), so size must come from the Cloudflare API, not from SQL.