fix(ci): rescue daily-regen from GitHub's silently dropped cron ticks

.github/workflows/daily-regen.yml

@@ -4,21 +4,31 @@ run-name: "Scheduled regen (${{ github.event.inputs.specification_id || github.e
 # Picks the N oldest specs (by most-recent implementation `updated` timestamp)
 # and re-dispatches `bulk-generate.yml` for each. Default N=1 per cron tick.
 #
-# Schedule: hourly, skipping the 20:00–23:59 Berlin (CEST) evening window.
-# Berlin CEST evening hours 20, 21, 22, 23 map to UTC 18, 19, 20, 21 — these
-# are intentionally skipped so runs never start during the user's evening.
-# All other UTC hours run, giving 20 ticks per day.
+# Schedule: every 2 hours, skipping the fixed 18:00–21:59 UTC window so runs
+# never start during the user's evening. The window was sized for Berlin
+# summer time (CEST, UTC+2 → 20:00–23:59 local); Actions cron is always UTC,
+# so under winter time (CET, UTC+1) it covers 19:00–22:59 local instead —
+# accepted drift, not a bug. That gives 10 ticks per day.
+#
+# The tick fires at minute 17 (not :00) on purpose: GitHub's scheduler is
+# most overloaded at the top of the hour and delays/drops `schedule` events
+# then (github.com docs on `on.schedule`). This repo has seen multi-day
+# streaks of silently dropped ticks for exactly this workflow while
+# less-frequent crons kept firing. As a second line of defence,
+# watchdog-stuck-jobs.yml re-dispatches this workflow when main has seen
+# no new run for >10 h outside the quiet window.
 #
 # bulk-generate is serialised via its own concurrency group. Default model is
 # Sonnet for higher implementation quality; can be overridden per manual run.
 #
 # Triggers:
 #   - schedule: 10× daily (UTC, every 2 hours, skipping the 18–21 UTC window) → Sonnet
 #   - workflow_dispatch: manual, with inputs for spec id, model, count, dry-run
+#     (also used by the watchdog's cron-liveness rescue, with all defaults)
 
 on:
   schedule:
-    - cron: '0 0,2,4,6,8,10,12,14,16,22 * * *'
+    - cron: '17 0,2,4,6,8,10,12,14,16,22 * * *'
   workflow_dispatch:
     inputs:
       specification_id:

.github/workflows/watchdog-stuck-jobs.yml

@@ -1,7 +1,7 @@
 name: "Watchdog: Stuck Jobs"
 run-name: "Watchdog: scan ${{ github.event_name == 'workflow_dispatch' && '(manual)' || '(cron)' }}"
 
-# Periodic safety net for the impl pipeline. Catches three failure modes
+# Periodic safety net for the impl pipeline. Catches four failure modes
 # the regular workflows miss:
 #   1. PRs labeled `ai-review-failed` without `ai-review-rescued`
 #      (impl-review-retry.yml normally handles these; watchdog covers
@@ -10,6 +10,8 @@ run-name: "Watchdog: scan ${{ github.event_name == 'workflow_dispatch' && '(manu
 #      (review handed off to repair, repair crashed, nothing else fired).
 #   3. spec-ready issues with `generate:<lib>` or `impl:<lib>:failed` and
 #      no open PR for that (spec, lib) pair after the staleness window.
+#   4. daily-regen's cron silently starved by GitHub's scheduler (no run
+#      for >10 h outside the quiet window) — re-dispatched manually.
 #
 # Retries are bounded per-cause via marker labels:
 #   - `ai-review-rescued`            (review failures)
@@ -295,4 +297,46 @@ jobs:
             done
           done < <(echo "$ISS_JSON" | jq -c '.[]')
 
+          ##### C) Cron-liveness: rescue a starved daily-regen schedule ######
+          # GitHub silently drops `schedule` ticks for daily-regen (the
+          # repo's most frequent cron) during scheduler overload — observed
+          # as multi-day gaps while this watchdog and bot-serving-check kept
+          # firing. daily-regen ticks are ≤6 h apart (2 h cadence plus the
+          # 18–21 UTC quiet window), so a >10 h silence means the schedule
+          # is starved; re-dispatch it manually with default inputs.
+          # Hours 17–21 UTC are skipped so the rescue never launches a
+          # pipeline into the fixed 18–21 UTC quiet window the cron itself
+          # deliberately avoids (17 included because bulk-generate starts
+          # immediately after dispatch and would spill into it). Like the
+          # cron, the window is UTC-fixed: it matches Berlin evening under
+          # CEST and sits an hour earlier in local terms under CET.
+          LIVENESS_HOURS=10
+          HOUR=$(date -u +%-H)
+          if (( HOUR >= 17 && HOUR <= 21 )); then
+            echo "::notice::daily-regen liveness: inside/adjacent to quiet window (UTC hour $HOUR) — check skipped"
+          else
+            # --branch main: schedule runs (and rescue dispatches) live on the
+            # default branch; a manual run on a feature branch must not mask a
+            # starved main schedule. createdAt (not startedAt): always set,
+            # even while the newest run is still queued.
+            LAST=$(gh run list --workflow daily-regen.yml --branch main --limit 1 \
+              --json createdAt --jq '.[0].createdAt // empty')
+            if [[ -z "$LAST" ]]; then
+              echo "::warning::daily-regen liveness: no runs found on main — skipping rescue"
+            else
+              last_sec=$(date -u -d "$LAST" +%s)
+              # Fresh timestamp: NOW from the top of the script is stale by
+              # however long the A/B scans took.
+              now_c=$(date -u +%s)
+              gap=$(( now_c - last_sec ))
+              gap_hm=$(printf '%dh%02dm' $(( gap / 3600 )) $(( (gap % 3600) / 60 )))
+              if (( gap > LIVENESS_HOURS * 3600 )); then
+                echo "::warning::daily-regen: no new run on main for ${gap_hm} (> ${LIVENESS_HOURS}h) — schedule starved, re-dispatching"
+                dispatch "daily-regen: cron starved (${gap_hm} silent)" daily-regen.yml
+              else
+                echo "daily-regen liveness: newest run on main created ${gap_hm} ago — healthy"
+              fi
+            fi
+          fi
+
           echo "::notice::Watchdog scan complete"

CHANGELOG.md

@@ -129,6 +129,14 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
 
 ### Fixed
 
+- **Scheduled implementation regeneration is harder to starve silently** — `daily-regen.yml`'s
+  cron had gone silent for multi-day stretches (a mix of GitHub's documented top-of-hour
+  scheduler overload and the workflow being manually disabled), and because every run that did
+  start was green, nothing alarmed. The cron now fires at :17 instead of :00, and
+  `watchdog-stuck-jobs.yml` gained a cron-liveness rescue that re-dispatches daily-regen
+  whenever main has seen no new run created for >10 h outside the Berlin-evening quiet window
+  (a manually disabled workflow stays disabled — the rescue cannot and does not re-enable it),
+  capping scheduler-side starvation at typically ~half a day instead of weeks (#9649).
 - **The whole Claude pipeline generates again — CI runs pass the repo's permission allowlist
   explicitly** — since the July 5 `claude-code-action` bump (bundled CLI 2.1.170 → 2.1.195, by
   now 2.1.211), Claude Code ignores `permissions.allow` from a committed `.claude/settings.json`

docs/workflows/overview.md

@@ -167,7 +167,8 @@ Located in `.github/workflows/`:
 | `impl-repair.yml` | Fixes rejected implementations |
 | `impl-merge.yml` | Merges approved PRs |
 | `bulk-generate.yml` | Batch implementation generation |
-| `daily-regen.yml` | Cron-driven regeneration of the oldest implementations |
+| `daily-regen.yml` | Cron-driven regeneration of the oldest implementations (10×/day at :17 past UTC hours 0–16 even + 22, dodging GitHub's top-of-hour scheduler overload and the 18–21 UTC quiet window) |
+| `watchdog-stuck-jobs.yml` | 6-hourly safety net: re-dispatches stuck reviews/repairs/merges/generations and rescues daily-regen when its cron is silently starved by GitHub (>10 h without a run) |
 | `report-validate.yml` | Validates user-submitted issue reports |
 | `sync-postgres.yml` | Syncs `plots/` filesystem state to PostgreSQL on push to main |
 | `sync-labels.yml` | Auto-syncs spec/impl labels after manual PR merges |