fix(drift): repair broken weekly Drift Watchdog — 100% false-positive issues since May 25#116
Open
helloiamvu wants to merge 1 commit into
Open
fix(drift): repair broken weekly Drift Watchdog — 100% false-positive issues since May 25#116helloiamvu wants to merge 1 commit into
helloiamvu wants to merge 1 commit into
Conversation
capture_drift.py called `mostlyright.research(...)` without importing `mostlyright`, so every weekly cron NameError'd on all 5 cases, wrote zero captures, and soft-exited 0. compare.py then reported every case as "drift capture missing", wrote drift-report.md, and the workflow opened a GH issue from it — a 100% false-positive "Drift detected" issue every Monday (#6, #56), never once a real comparison. - capture_drift.py: add the missing `import mostlyright`. - compare.py: split genuine value/dtype/shape drift (opens an issue) from missing captures (capture/network/infra failure -> stderr warning, never written to the report, never opens an issue). If every capture is missing, no report is written. main() now takes optional parity_dir/drift_dir for testability; the CLI path (`python compare.py`) is unchanged. - test_drift.py: regression tests covering missing-capture (no report), real value drift (report written), matching (no report), and that capture_drift imports mostlyright. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Parity ticket gate: PASSED See |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every Monday since 2026-05-25 the Drift Watchdog cron has opened a false-positive "Drift detected" issue (#6, #56, #73, #77, #85, #86, #105) without ever running a single real comparison.
Root cause:
tests/fixtures/drift/capture_drift.pycallsmostlyright.research(...)but never importsmostlyright. Every weekly run NameErrors on all 5 cases, writes zero captures, and soft-exits 0.compare.pythen reports every case as "drift capture missing", writesdrift-report.md, and the workflow opens a GH issue from it.Fix
import mostlyright.main()now takes optionalparity_dir/drift_dirfor testability; the CLI path (python compare.py) is unchanged.Originally authored 2026-06-04 against v1.5.2; rebased clean onto current main (v1.15.0) — the drift fixture scripts were untouched in between.
tests/test_drift.pypasses against the v1.15.0research(); full fast suite + typecheck green via pre-push hook.The TS twin (
drift-rotate-ts.yml/drift_capture.ts) importsresearchcorrectly and does not share this bug.TS Parity
N/A — CI/test-fixture fix only; no public API surface change.
Fixes #73. Fixes #77. Fixes #85. Fixes #86. Fixes #105.
🤖 Generated with Claude Code