feat(pr-workflow): add the /mms-evidence orchestrator, runners, and publish gate - #84
Draft
MajorLift wants to merge 49 commits into
Draft
feat(pr-workflow): add the /mms-evidence orchestrator, runners, and publish gate#84MajorLift wants to merge 49 commits into
/mms-evidence orchestrator, runners, and publish gate#84MajorLift wants to merge 49 commits into
Conversation
Replaces #56, whose skill.md shipped frontmatter with no body at all — the installer's bodyAfterFrontmatter() returned empty, so an agent loading it got a description and six references it had no instruction to read. pr-validate: for a PR's specific falsifiable claim, name the observation that would prove the claim false, gather it, and publish it into the PR body. Drives the AEP harness (visual_validation, perf_validation) as the primary engine, backed by a catalog of complementary lanes, a trustworthiness gate that rejects vacuous passes, and a publishing flow with an audience-reachability rule for re-hosted artifacts. falsifying-test: the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. Its own falsifier is a base-commit failure for the wrong reason (import error, missing fixture, unrelated red), which looks identical in an exit code and proves nothing. pr-validate calls it as the engine behind lane B3. hooks/pr-evidence-gate.py enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable ref, a dump-as-resolver, a link-only or data-only exhibit, or a step waiver. It polices `gh api` body writes as well as the porcelain, since a PATCH to a comment is the same publish with a different spelling.
Over-scrubbed. The audience is the MetaMask org, and the ticket is the evidence for the claim the item makes — that the LaunchDarkly provisioning blocker covers only the prod-flag half of that lane. Without it the example is an assertion. The scrub line is personal references, not org-internal ones.
MajorLift
changed the title
pr-validate and falsifying-test skills
MajorLift
marked this pull request as draft
July 30, 2026 14:03
`pr-validate`'s other engines are placed by subject — `react-render-proof` in `performance`, `memory-leak-hunt` in `stability`, `supply-chain-audit` in `security`. This one was placed by its caller instead. Writing a test that fails on the base commit and passes on the branch is a testing technique, and `testing/` already holds techniques of that kind (`e2e-flakiness-patterns`, `test-layer-placement`, `performance-testing`), while `pr-workflow/` is uniformly PR-lifecycle stages. Both references to it are by name rather than path, so nothing needed updating.
MajorLift
marked this pull request as ready for review
July 30, 2026 18:15
`D6` covers claims where a PR hand-writes an artifact that restates an existing source — a type, schema, vendored constant, or checked-in policy. Both arms sit at the same commit and differ by a substitution rather than a ref, so there is no build or merge boundary to confound the result.
Contributor
Author
Context budgetWhat this PR costs an agent, measured from an install rather than read from the diff. Three tiers, and only the first is unavoidable.
Frontmatter is the only tier paid unconditionally — every agent loads it on every run once the skill is installed, used or not, because it is what the agent reads to decide relevance. The 28 skills across the eleven open skill PRs sit at a median of ~1,716 tokens selected and ~1,860 with references followed. All are within the 1,536-character description budget. Selected is paid only when the agent picks the skill. + refs & knowledge is the ceiling if every bundled reference is then read; it is a worst case, not an expectation. Method
These figures are pinned to the commit above and drift on every push; #96 tracks automating them. |
`pr-validate` cost ~9,059 tokens the moment an agent selected it — 5.6× the median of the 26 skills in the open PRs — and a quarter of that was the AEP local-run procedure, which most validations never touch. The skill's own Sufficiency section tells you to prefer a lighter lane; the body charged you for the heavy one regardless. Preflight, run mechanics, and teardown move to `references/aep-local-run.md`, which the body already linked twice and which did not exist. The link was dangling — the same defect class the `knowledge/` guard catches, on a path nothing checks. Publishing keeps the decisions (surface by ownership, post complete once, falsifier-forward, scrub) and points at `references/evidence-publishing.md` for the mechanics it already documents in full. Body 34,431 → 25,736 bytes, so a selected skill is ~6,812 tokens installed rather than ~9,059. Nothing is lost: it sits behind the same progressive disclosure boundary as the other seven references, read when an AEP run is actually warranted. Description trimmed 1,147 → 885 characters. It was over the 1,024 ceiling that #47 enforces, so it would have failed that check on merge.
Trimmed to 885 characters to fit a 1,024 budget that turned out to be unverified — no operator observed rejects or truncates a longer description, and several over 1,024 load today. Back to 1,147, under the 1,536 budget. The description is the discovery surface, so the 262 characters were trigger cues: the subcommand list and the phrasings that route a request here rather than to another skill. Cutting them made the skill harder to select, which is a functional loss and not a cosmetic one. The body restructure is unaffected — that removed duplication behind a reference, which costs nothing at selection time.
2af1db1 carries the reasoning but not the change: its `git add` was chained into a command the commit guard rejected, so the commit was created from an already staged tree and landed empty — same 885-character description as its parent. This applies it. Back to 1,147 characters, under the 1,536 budget. The 262 characters are trigger cues — the subcommand list and the phrasings that route a request here rather than to a sibling skill — so losing them made the skill harder to select, which is a functional loss rather than a cosmetic one.
Two of the three assumptions in the old name were false. The skill runs in the inner loop against uncommitted changes, and on a symptom with no claim and no PR at all, so `pr-` narrowed it to one of three modes. `validate` named an activity where the deliverable is an artifact, and read as a checklist exercise — the posture the skill spends its opening section arguing against. `evidence` names what it produces, and covers a refutation as naturally as a confirmation. The verdicts are proven, refuted, and inconclusive; a name promising proof would make two of those read as failure. The description is rewritten rather than search-replaced. It now states all three modes, since the old one described only the PR case and so under-selected for the other two, and it names the trigger as `mms-evidence` — the form the installer actually emits. Eleven skills across the repo still promise the unprefixed `/<name>` in their descriptions while installing prefixed; this corrects the one being renamed.
MajorLift
changed the title
pr-validate and falsifying-test skillsevidence and falsifying-test skills
MajorLift
changed the title
evidence and falsifying-test skills/mms-evidence orchestrator and falsifying-test skill
The engine table still routed the memory-leak category to `/memory-leak-hunt`, which was renamed to `/memory-leak` — the rename missed the PR that performed it. Two `[[snake_case]]` entries were wiki links to a private authoring vault; they render as literal brackets here and resolve for no reader. One had a real counterpart in `references/` and now links it; the other pointed at a file that does not exist and is dropped rather than left dangling. `falsifying-test` named its evidence categories as `B3` and `B7`. Those are addresses into `evidence-catalog.md`, not names, so both now use the category name and link the catalog.
`race-condition-proof` is now `race-condition-repro` and `react-render-proof` is now `react-render-delta`. Both are named here as engines, in the catalog, the engine table, and the sibling reference — none of which the renaming branches can reach.
The catalog is 41 full lane specs with no summary, so a reader arriving from a link lands mid-document with no way to see the shape of it. Adds a generated "Lanes at a glance" table: family, count, and every lane id with its title. Two placement bugs: - `C9` sat inside the `# D. Build output` section, so scanning family C missed the lane backing `memory-leak`, and scanning D found a stranger. - `B7` sat between `B3` and `B4`. All 41 lanes now read in order. Also removes six pointers into a private authoring vault — four inline `exogram` references and two full `exogram-daemon/...` paths. They resolve for no reader of a public repository. Every substantive claim they were attached to is kept; only the dangling pointer is dropped. `memory-leak-hunt` updated to `memory-leak`.
The catalog had 41 lanes and none for how long a build takes. Family D covered build *output* — size, chunks, policy, permissions, variants — and `C5` covers runtime, so a toolchain change had no category to publish into even though a skill for measuring one is specified in #102. `D7` is the dev-loop half: paired A/B, cold and warm as separate numbers, with the four confounds that each return a favourable result when uncontrolled — warm cache leaking into the cold arm, worker-pool startup amortised away, core count that does not transfer off the measuring machine, and watch rebuilds presented as cold builds. `G6` is the CI half, in family G because its dominant confound is a process one: `get-requirements.yml` skips jobs when build output matches base, so a measured speedup is often a skipped job. Family D is retitled from "Build output" to "Build", since it now covers both. The at-a-glance index is regenerated rather than hand-patched — it is derived from the headings, and hand-editing it is how it drifts.
This was referenced Jul 31, 2026
An 18-comment trial run met none of this skill's output requirements. The cause was structural, not behavioural: `VALIDATION_RUN_START` and the in-situ capture rule occurred zero times in skill.md and only in a reference costing ~5x the body to open, described there as "image re-hosting and the privacy scrub". The publish gate checks none of them either. All three layers failed open. Moves six non-negotiables and the canonical output shape into the body, where they load with the work: 1. Ship an artifact the reader can check without trusting you. Pasted terminal text is indistinguishable from invented terminal text — running the check justifies your belief, not the reader's. 2. `proven` requires execution; reading gives shape, never power. Run arm B against your own probe: one that passes with the mechanism deleted is measuring something else. 3. No "what would close it" section — that is an unfinished run formatted to look finished. Imperative-mood prose means the artifact does not exist. 4. Write to the reviewer who arrives, not whoever commissioned the run. 5. Delete findings whose entire content is test quality, unless critical. 6. Route privacy and security findings to the private tracker. Derived from eight postmortems in exogram-core; the reference keeps the full recipe.
A recipe returns as many answers as it has operators, and any output an operator
retypes carries the operator's provenance rather than the measurement's. This
ships the mechanism instead.
Runs arm A, mutates one line, runs arm B, restores the source, and writes
`falsify-<label>.{json,md}` plus both raw logs itself — nothing is transcribed.
The exit code is the verdict, so CI gates on it directly: 0 falsifying, 1 vacuous,
2 arm A already failing, 3 usage error.
Every artifact pins HEAD, node version, yarn.lock hash, and the tracked-change
count, so two operators either produce comparable results or visibly do not.
Verified against both outcomes on metamask-extension at 796685ce7b7: the perps
coalescing suite reports `falsifying` (10 passed, 2 failed under mutation), and
the token-search suite reports `vacuous` (3 passed both arms, so it does not
test the abort it appears to test).
…o retype them `retention-scan.py` and `policy-audit.py` already do the analysis well; both print to stdout, which makes the operator the capture device and returns provenance to whoever pasted the output. This wraps any command so the tool writes the artifact. Emits <label>.log verbatim, plus .json and an attachable .md that quotes the log rather than summarising it, with HEAD, tracked-change count, node, python, and yarn.lock hash pinned in each. The wrapped exit code passes through for CI. `--verdict` is stated by the caller, never inferred from the exit code. The first run of this script proved why: it labelled a policy audit "pass" while the output listed sixteen newly granted capabilities, because policy-audit.py exits 0 regardless. With no --verdict it now says "ran to completion, no verdict asserted". Verified on both scripts against real PRs, including a deliberately wrong invocation — which produces an artifact containing the traceback rather than a fabricated finding.
A memoization claim is a claim about a count, and reselect publishes the count. This generates a probe, runs it, deletes it, and writes the artifact, so the number never passes through an operator's hands. Three conditions; the middle one discriminates. A selector built on narrowed input selectors is unmoved by a write it does not read, while one taking `state.metamask` wholesale recomputes on every unrelated write in the app. Verified against both shapes on main, so the runner is shown to distinguish them rather than only to report success: getWalletsWithAccounts 1 / 1 / 6 narrowed selectRampsControllerState 1 / 6 / 11 recomputes on unrelated writes That completes runner coverage for the catalog lanes with engine skills: B3 and B7 via falsify-probe, C4 here, C9 and D3 via capture around their existing analysis scripts.
Whether a hand-written type agrees with the source it restates is a question only the compiler can settle. Arm A typechecks the baseline, arm B applies the substitution, and the finding is the error diff. Source is restored on exit, including on interrupt. Carries the warning the lane most needs: a silent arm B is not proof of agreement. Indexing and `.match()` compile against `string` and `string[]` alike, so without `--probe` injecting a deliberately-typed sink, the lane reports false clean on exactly the divergence it exists to find. When arm A already fails it stops and says nothing was established, alongside the module/export error count — but it does not classify from that ratio. The first real run had 124 of 280 errors as install artifacts while tripping no majority rule, because other codes are downstream of the same missing types. A threshold there would be a number I could not justify, so it reports the breakdown and leaves the judgement with the operator.
Gating on arm A being error-free was wrong: the finding was always the diff, so a single unrelated pre-existing error — a local work-in-progress file, in the run that exposed this — vetoed the whole lane. Baseline errors are now subtracted and only errors new under substitution count. Verified end to end on shared/lib/transactions-controller-utils.ts, where the local `LogWithTopicsArray` declares `topics?: string[]` against an upstream `topics?: string`: substitution alone 1 -> 1 errors, 0 new silent substitution + typed sink 1 -> 2 errors, 1 new TS2322 at the sink That is the lane's central caution demonstrated rather than asserted. Indexing and `.match()` compile against both shapes, so the obvious probe reports a false clean; only a deliberately-typed sink surfaces the divergence. A silent arm B means the probe was too weak, not that the types agree. Also fixes a `grep -c ... || echo 0` double-fire that produced "0\n0" and an integer-comparison error — the same shape already fixed once in this script's tracked-change count.
…lished Phase 0 of the /attest command. Everything greppable is checked before a model is asked for judgement, because a model asked "is this good evidence?" answers from inside the frame that produced the text. Marker pair, canonical header, verdict line, environment pin, a captured artifact, no "what would close it", no first-person process narration, and `proven` only where an execution artifact exists. Check 5 carries the weight: if every character of the output is one the operator typed, the run published an assertion. `--reference` compares capture density against a known-good artifact. Verified in both directions. A retracted run-1 comment is BLOCKED on three checks — no captured artifact, a "what would close it" section, and an unearned `proven`. A runner-produced artifact passes all eight. Building it reproduced two bugs it exists to catch: `hasre -i '<pat>'` passed `-i` as the pattern, so three checks silently grepped for the literal string and returned false passes; and an over-escaped backtick made the environment-pin check never match. Both found by running the gate against a file whose expected verdict was already known.
7 tasks
Check 5 has been rewritten four times today and each time it tested a property of the plaintext — a provenance marker, a placeholder in the command, a local path. Each caught one defect and missed the next, because every property of text is forgeable by whatever emits the text. Four runs shipped through it. It now requires a capture the reader verifies without going through the author: an image of the tool's surface, a link that re-executes, or a hosted artifact. A fenced block sits beside one of those and is never the evidence itself.
A local run's only witness is its author, so it cannot satisfy the requirement that a reader verify without going through them. Every defect class this suite has shipped was a local-environment one: a helper in `/tmp`, a probe deleted after the run, an absolute path, a drifted toolchain, a contended host whose numbers were published and retracted. None is expressible in CI. Two controls ride along because both were learned the hard way. A `baseline` input: twice a run reported no finding when what it lacked was a comparison. A determinism check: the head arm runs twice and the numbers are not endorsed if they move. Inputs reach the shell through `env`, never spliced into the script text.
The workflow fetched the runners from the skills repo's `main`, where they do not exist — they are on this branch. Every dispatch would have failed at that step, and a sparse checkout of an absent path succeeds with an empty directory, so the failure would have surfaced two steps later as "No such file" with no hint that the ref was the cause. Pinned to a commit for the same reason the measured ref must be, with an explicit check that names EVIDENCE_SKILLS_REF as the thing to change.
…all framing Three corrections to the runner workflow, all found by pointing it somewhere. `target_repo` is required with no default: its stated purpose is to be proved somewhere harmless first, and a default pointed the first dispatch at the repo under review. `needs_install` skips a ten-minute yarn install for runners that only read files. And the header still described installing this into the repo under review, which stopped being true once the target became an input — that stale sentence is what made a fork of the extension look necessary.
Putting the showcase runs and the trial runs in one table shows the causation running the wrong way: eight of eleven hand-built runs attach a capture a reader can open, against one of twelve built by these runners. Nothing was neglected. A runner emits clean stdout, clean stdout formats beautifully into a fenced block, and a fenced block looks like evidence — so automating the measurement automated away the part that made it checkable. The hand-built runs had no such thing to reach for and went and got a real capture. Each runner's provenance line now prints the run URL under CI, and under a local run prints that there is no reader-verifiable capture and the workflow should be used before publishing. The confession belongs in the exhibit, not in a gate that has to remember to look for it.
"Plaintext is the wrong medium" was too broad. The rule is about where verification routes, and a line-level permalink routes it away from the author exactly as an image does — the reader clicks and sees what you saw. That is the normal case for the audit lanes, whose findings are facts about code that exists rather than results of running something. A screenshot of a policy diff is less checkable than a permalink to it, not more. The bar there is comprehensive linking: a call site per grant, a re-runnable search per claimed absence, a file and line per version claim. A row naming a package and a capability with no link is a claim on the author's word in a technical register.
Four corrections, each from a run that failed rather than from review. `corepack enable` before `setup-node`: a target pinning its package manager via `packageManager` makes setup-node's `cache: yarn` probe run under the runner's global yarn 1.22, which refuses — every jest runner died at setup with nothing measured. `logs` added to the determinism exclusions, since `render-count` is the one runner writing the plural key and so failed that check on every run while reporting identical counts; a warning always wrong for one runner teaches its reader to publish through it. A `probe_path` input, because `render-count` takes a hand-written probe and a probe living only on the author's disk is the exact defect a run URL exists to remove. And `skills_repo`/`skills_ref`, because a runner fix and the run that needs it cannot both wait on a review. The sparse checkout now also pulls `domains/security/skills`, so the analysis scripts wrapped by `capture` are on disk without being runners themselves.
…wice A base arm and a head arm both printed `yarn jest <probe>`, leaving the reader no way to tell which commit produced which number. On a clean result, where the two arms agree, the block reads as a single measurement printed twice. The publish gate fails a pair of identical `$` lines for exactly this reason and caught it on a real comment. Fixing the check's input rather than the check.
This was referenced Aug 2, 2026
Merged
Merged
First CI run of this runner, and it reported `divergence surfaced` with six new errors — all TS1109/TS1011/TS1128, the syntactic family. The substitution had landed one line above the type declaration and broken parsing, so arm B never type-checked at all. By error count that is indistinguishable from the local type genuinely disagreeing with its authoritative source. `falsify-probe` has carried the equivalent guard since a syntax-breaking mutation looked like a falsification. This runner shipped without one and had never executed in CI, which is where the gap surfaced.
This was referenced Aug 2, 2026
Four runs measured `$SHA^..$SHA` on branches of five to twenty-nine commits and produced clean artifacts for a fraction of each change — same runner, same green run, nothing in the output distinguishing it from a finished measurement. So the range is now a non-negotiable, with the compare endpoint's `merge_base_commit.sha` named because `.base.sha` is the base branch tip and moves. Two rules alongside it, from the same batch: what a run could not see is a finding to state rather than a gap to omit, and the label on a number is caller-stated for the same reason the verdict is — a probe counting distinct context values published under a fixed "renders" heading passes every check while naming the wrong quantity.
…on it took A mutation runner echoed its `--replace` argument into the artifact, so the two could never disagree. They did: an `awk -v` assignment escape-processed the value and wrote a different line than the one requested, narrowing a regex meant to be widened. The suite ran the same test count in both arms, a different test failed than the one targeted, and the run reported power over a mechanism it never touched. Nothing in the artifact could have shown it.
MajorLift
marked this pull request as draft
August 3, 2026 09:42
MajorLift
added a commit
that referenced
this pull request
Aug 3, 2026
Each encodes a failure that actually shipped, rather than a principle that sounded right. `unmeasured-join` targets the costliest one — a conclusion assembled from true facts through a step nobody measured, which survives review because the facts check out and the join reads as prose. `instrument-check` requires a positive and a negative control before a measurement counts, after a mutation runner reported a falsification for a line it never wrote. `scope-of-search` makes a negative carry the pattern that produced it, after a grep keyed on a variable name declared absent what its own output printed two blocks later. `coverage-partition` replaces "the test has power" with which cases guard which mechanism. `selection-audit` separates a count from the rule that produced it, after two different selections landed on the same number and the match closed the question. `distinguishing-observation` and `observability-gap` point the same discipline at debugging: design the observation that separates candidates rather than confirms the favourite, and establish what signal exists on a path before reading more of it. The `evidence` links are forward references — that skill ships in #84 and is not on main yet.
4 tasks
Step 6 asked for both runs shown and left the operator to produce them, which is the shape that yields a retyped paste indistinguishable from output nobody ran. The run workflow already takes `ref` and `baseline`, executes the same command at both commits, and attaches the artifacts to a URL a reader can open — so the step names it. The larger risk was confusion with `falsify-probe.sh`, which shares the two-arm shape and answers a different question: base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether it notices the mechanism going away. A test can pass either and fail the other. The skill now states the distinction in a table rather than leaving two similarly named things to be conflated, and notes that the runner's guards mechanise this skill's own falsifier — a red arm that ran fewer tests, or failed to load, is refused rather than counted.
Two things in this PR shared a root and a two-arm shape while answering different questions: the skill compares a base commit against a branch to show a test is connected to the reported bug, and `falsify-probe.sh` mutates one line at one commit to show a suite notices the mechanism going away. A test can satisfy either and fail the other, so the names had to stop rhyming. `red-on-base` names the skill's own discipline — the test must be red on the base commit — and shares no root with the runner. The evidence category keeps its name: a falsifying regression test is the artifact, and this is the procedure that produces one. The runner's stderr described its own result as "a falsifying test", which was the same collision inside the output of the thing causing it; it now says what it proves and points at the other experiment by name.
MajorLift
changed the title
/mms-evidence orchestrator and falsifying-test skill/mms-evidence orchestrator and red-on-base skill
Every other engine `evidence` names ships in its own pull request — `memory-leak`, `race-condition-repro`, `supply-chain-audit`, `react-render-delta`, `agent-run-cost`. This one was the sole exception, and the reason was authoring order rather than design: it was written alongside the orchestrator before the split between the instrument and the reasoning that points it existed as a concept. It lands in #106 with the other reasoning skills, whose substance is the same kind — what counts as proof, and how a proof can look right while testing the wrong thing. What stays here is the machinery: the runners, the run workflow, the gate, the hooks. The B3 engine cell now names a skill that arrives in #106, which is a dangling name in a table rather than a broken link, and resolves whichever order the two merge.
MajorLift
added a commit
that referenced
this pull request
Aug 3, 2026
Its substance is the same as everything else here: what counts as proof, and how a proof can look right while testing the wrong thing. Its falsifier — a base commit that fails for the wrong reason produces an identical exit code and proves nothing — is the same move `unmeasured-join` and `scope-of-search` make on different material. It sat in #84 because it was written alongside the orchestrator, before the instrument and the reasoning that points it were separate ideas. Every other engine already ships in its own pull request. Renamed from `falsifying-test` before the move, because it and `falsify-probe.sh` shared a root while answering different questions — base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether a suite notices the mechanism going away.
MajorLift
changed the title
/mms-evidence orchestrator and red-on-base skill/mms-evidence orchestrator, runners, and publish gate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
evidencetopr-workflow: the orchestrator, six runners, the run workflow, the publish gate and the hooks.evidenceA reviewer takes the author's tested locally, works on trust. Capture skills replay a flow and photograph the screen — but a clean screenshot is also what a silently-swallowed failure produces.
pr-readiness-checkproves tests are present, not that behaviour is correct.evidencenames the observation that would prove the claim false, goes looking for it, and publishes what it finds where review happens.Given / when / thenplus an explicit falsifier.curlbefore shipping. A link only its author can open is not evidence.What changed since this PR was opened: the evidence had to stop being text
The skill originally shipped guidance. Guidance is not what failed.
Twelve validation runs produced by this skill were published to
metamask-extensionand thenretracted the same day. Every one had passed the gate. The gate's checks were all properties of
the plaintext — does a verdict line exist, is an environment pinned, is a falsifier named —
and every property of text is forgeable by whatever emits it. A runner writes clean stdout,
clean stdout formats into a fenced code block, and a fenced code block looks like evidence.
The findings were not withdrawn; the medium was.
So the rule the skill now enforces is about medium rather than wording: cite what exists,
capture what you ran. A reader has to be able to check the measurement without going through
the author, which text pasted by the author cannot provide no matter how it is formatted. The
one exception is line-level citation to something externally verifiable — a permalink resolves,
so an audit built from permalinks is checkable even though it is prose.
Two consequences make up most of the diff added since:
The measurement moved to CI.
assets/evidence-run.ymlruns any runner against any target repository at a pinned SHA. A local run's only witness is
its author;
actions/runs/<id>is itself the artifact. Every failure class this suite hasshipped was a local-environment failure — a helper script in
/tmp, a probe deleted after therun, an absolute path, a contended host producing numbers that had to be retracted — and none
of those is expressible in CI. The workflow lives in one repository and measures others, so
nothing needs installing in the repo under review.
Six runners write the artifact, so nothing is retyped.
falsify-probe.shselector-recompute.shrender-count.shtsc-substitution.shcapture.shattest-gate.shEach emits JSON, a log, and a markdown block, and each says in its own footer whether a reader
can verify it. Run locally, that footer reads "no reader-verifiable capture" — the artifact
discloses its own weakness rather than leaving it for a gate to catch.
The gate now tests the medium
attest-gate.shruns 11 checks:marker pair,canonical header,verdict line,environment pinned,captured artifact,no prescriptions,no process narration,verdict is earned,verdict matches artifact,floats something for review,disclaimer present and early.Check 5 is the one that matters and it was rewritten four times. It now requires an image, a
run URL, a gist, or a hosted log — and then rejects three specific forgeries that passed
earlier versions: a placeholder in a shown command, a local path in a shown command, and an A/B
pair rendered as one command line printed twice with different output.
Nine non-negotiables, in the skill rather than a reference
A requirement you have to fetch is advisory, so these sit in
skill.md. Four were added byruns that failed while this PR was open:
$SHA^..$SHAon atwenty-six-commit branch is a twenty-sixth of the change and looks exactly like a finished
measurement — same runner, same green run, same artifact.
is not in the measurement, and saying where the edge is is the finding rather than an
apology for one.
probe printed; it knows the field's name, not what was counted. A probe counting distinct
context values, published under a fixed heading of "renders", is a correct measurement of the
wrong quantity with every check passing.
that echoes its own argument into the artifact cannot detect its own misfire. One did: an
awk -vassignment escape-processed a replacement, wrote a different line than requested,failed a different test than targeted, ran the full suite, and reported a falsification for a
mechanism it never touched.
The evidence catalog — 41 lanes across 7 families
The substance of
evidenceis the taxonomy, which lives inreferences/evidence-catalog.md(commit-pinned, 34 KB). The table below summarises it so the shape is reviewable without opening the file. Each family answers a different kind of question, so the family follows from what the claim asserts.A1 visual_validation— before/after of a reachable surfaceB3 falsifying test— fails on base, passes on branchC4 render & selector proof— WDYR counts,.recomputations()D3 LavaMoat policy diff— newly granted capabilitiesE1 Sentry links— before/after, in situF1 state migration—changedKeys, old→new stateG5 CI-workflow change— run on a test forkBolded lanes carry a claim on their own; the rest usually corroborate.
Ten skills in #106 depend on this one. The audit and reasoning skills there —
falsifiers-first,unintended-breakage,silent-failure,unmeasured-join,instrument-check,scope-of-search,coverage-partition,selection-audit— referenceevidenceas their execution layer, and six carry relative links that resolve only once this merges. Four of the non-negotiables above were extracted by runs that produced findings those skills now exist to prevent, so the two PRs are the same body of work split by what it changes: this one changes the instrument, #106 changes the reasoning that points it.Five lanes have grown their own engine skills, which is why several open PRs exist:
B3→red-on-base(this PR),B7→race-condition-repro,C4→react-render-delta,C9→memory-leak,D3→supply-chain-audit→lavamoat-policy-diligence.A known gap: every family covers product behavior. None covers agent behavior — whether
a skill improves what an agent produces. Picking a lane cannot answer "does this guidance work",
which is why reviewing skills PRs is a different problem from reviewing code.
The engine moved to #106
red-on-base— the base-against-branch proof behind catalog lane B3 — shipped here until now, and was the only engine that did.memory-leak,race-condition-repro,supply-chain-audit,react-render-deltaandagent-run-costall ship in their own pull requests; this one was bundled because it was written alongside the orchestrator, before the instrument and the reasoning that points it were separate ideas.It was also renamed on the way out.
falsifying-testandfalsify-probe.shshared a root and a two-arm shape while asking different questions — base-against-branch asks whether a test is connected to the reported bug, one-commit-with-a-mutated-line asks whether a suite notices the mechanism going away — and a test can satisfy either while failing the other.What remains here is the machinery. The B3 engine cell names a skill that arrives in #106: a dangling name in a table, not a broken link, and it resolves whichever order the two merge.
Showcase —
red-on-baseapplied in this repoTwo PRs here were validated with it, so the output is checkable without leaving
MetaMask/skills.#94 — deliver knowledge a skill cites from another domain. Claim: four skills cite
testing-layers.md, which lives in another domain, so the reference cannot resolve for any consumer. Tests written against that claim, run at both refs:#87 — guard that installed knowledge references resolve. The falsifier the skill insists on is that a test failing on base for the wrong reason proves nothing — an import error and a real assertion exit identically. So the base failure is read, not just counted:
The message names the defect rather than a missing fixture, and the run reports 0 import or fixture errors. That check is the difference between a falsifying test and a test that merely goes red.
What it caught that a green suite did not. A description-budget check added during this work passed while asserting nothing: the regex anchored with
^and no multiline flag, so it matched zero files. Addingmthen made$match at every line end, catching the plain frontmatter form and silently missing the folded>-form — most of the corpus. Both passes looked identical in the runner. Only running the check against a known violation exposed it, which is the discipline the skill exists to enforce.hooks/pr-evidence-gate.pyA
PreToolUse:Bashhook that enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable/blob/<branch>/ref, a raw dump standing in as the resolver, a link-only or data-only exhibit, or a step waiver.Two properties worth calling out:
gh apibody writes, not just the porcelain (-f body=,-F body=@file,--input). A PATCH to a comment is the same publish with a different spelling; a porcelain-only matcher is a hole, not a gate.references/evidence-gate-setup.md.evidence-trustworthiness.mdstates plainly which items the hook enforces mechanically and which stay reader-applied (it sees vocabulary, not semantics — it cannot tell whether a screenshot actually shows the resolving UI).Files
domains/pr-workflow/skills/evidence/—skill.md, ninereferences/, sixscripts/,assets/evidence-run.yml,hooks/pr-evidence-gate.py; . Experimental. Commits GPG-signed.Merge after #99.
hooks/besideskill.mdis rejected by the structural linter until #99 adds it toBUNDLE_DIRS, so this fails CI onmaintoday through no fault of its own contents.Notes for reviewers
pr-workflowvstesting) and on AEP scope — the AEP repo is internal, and the skill points at its README for stack bring-up rather than reproducing setup here.evidenceis the heaviest skill in the catalogue at ~6,800 tokens once selected. The AEP run procedure sits behindreferences/aep-local-run.mdrather than inline, since most validations use a lighter lane.Validation runs
Trial runs of this PR's skills against merged
metamask-extensionPRs nobody flagged. Every claim was re-verified against the real diff before posting. Clean results are included on purpose — a skill that only ever reports problems cannot be calibrated.red-on-basered-on-baseEach comment carries a trial-run disclaimer and links back here for feedback.