Skip to content

Add typescript-compiler-blindspots skill - #85

Closed
MajorLift wants to merge 4 commits into
mainfrom
add-skill/typescript-type-proof
Closed

Add typescript-compiler-blindspots skill#85
MajorLift wants to merge 4 commits into
mainfrom
add-skill/typescript-type-proof

Conversation

@MajorLift

@MajorLift MajorLift commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Adds a typescript-compiler-blindspots skill under domains/testing.

The problem

A hand-written TypeScript type is a claim about a value's shape, and it compiles whether or not the claim is true. So a green tsc tells you the code is well-formed, not that the types are correct — and in a partially-migrated repo it often cannot tell you at all: with checkJs off, a type written for a function whose callers are still .js is checked against nothing and can drift indefinitely.

That is exactly the gap JS→TS migration PRs live in, and reviewing them by eye does not close it.

What the skill does

  • Makes the claim falsifiable with a two-arm substitution proof at a fixed commit — no rebase, no merge boundary:
    • Arm A — the PR as written. Must be silent; a noisy baseline means Arm B is unattributable and the run is inconclusive, not a pass.
    • Arm B — same tree, hand-written type swapped for the derived one, exercised as the real call site exercises it. Each new diagnostic is a disagreement the hand-written type concealed.
  • Gives a search order for the authoritative source — the call site (including a JSDoc @type on a JS caller, which sometimes already names it), SomeController['method'], a package already imported in the file, the sender of a message, ReturnType<typeof selector>, the @types/* declaration. Plus the case that matters for honesty: when no source exists, hand-writing is correct and gets recorded as a cleared falsifier.
  • Names the four divergence shapes that cover nearly every real finding — widening, dropped nullability, duplication, placeholder (Record<string, unknown>).
  • Audits the second axis: typing edits that quietly change runtime behavior — a literal swapped for a runtime enum lookup, a deleted default parameter, a call made optional so a throw becomes a silent no-op, a widened local propping up a check. Each with a reachability verdict rather than a "might be a bug."
  • Weights findings by observability — an unlikely failure inside a swallowed catch { captureException } can outrank a likely one in a loud path, because nothing goes red.

Contents

File
skill.md the method, the search order, the divergence taxonomy, the runtime-behavior pass
scripts/substitution-ab.sh runs both arms; refuses to read Arm B if Arm A is not silent
references/false-negatives.md the standing blind spots, each with a verified demonstration
references/worked-example.md 12 hand-written types on a real migration PR → 5 confirmed divergences, 5 cleared
references/metamask-extension.md repo notes: heap, probe location, checkJs off, authoritative-source lookup table

Added since first review: the standing blind spots

The skill originally covered one class — types hand-written to restate a source that already has one, found by substituting the derived type and diffing tsc output. references/false-negatives.md adds the class that needs no PR to find, because it is a property of the language and the config rather than of any diff:

  • Unsoundness: index access typed as if the element always exists, ?: accepting an explicit undefined, bivariant method parameters, covariant arrays, excess-property checks that fire only on fresh literals, structural typing erasing domain distinctions.
  • Boundaries: any absorbing any annotation, ambient declare module believed unconditionally, external data asserted rather than validated, JS callers checked not at all.
  • Config: skipLibCheck, exclude, include, checkJs — and the one worth saying out loud, that noEmit plus a transpiling bundler means a type error breaks a CI job, not the build.

Each is demonstrated, not asserted. The reference carries a file whose ten blocks are all genuinely wrong; typechecked against metamask-extension at 7fafda0 with that repo's own tsconfig.json, tsc reports zero errors on every one.

The rename follows from this: type-proof described what tsc already does. The subject is the complement — what it cannot do, because it checks declarations for internal consistency and never for correspondence to the source they restate.

Notes for reviewers

  • maturity: experimental — the method is proven on one PR (metamask-extension#44397), not yet across repos.
  • Extension specifics are in references/, not repos/, deliberately: a repos/ subdir containing only an extension overlay would make the skill skip installs for mobile and core. The technique applies to all three.
  • Review-side companion to the write-side guideline proposed in contributor-docs#178.
  • yarn test passes (41/41) and metamask-skills list --domain testing --maturity experimental discovers the skill.

The part I would push back on hardest

The worked example includes a claim the probes refuted — a provider return type I first read as unsound, which turned out fine once an unrelated nullability error one property ahead of it was isolated. It is in there on purpose: a substitution sweep that only ever confirms is indistinguishable from one that never isolated anything, so the skill requires reporting clearances alongside findings.

A hand-written TypeScript type is a claim about a value's shape, and it
compiles whether or not the claim is true — so a green build is not
evidence that the types are correct. In a partially-migrated repo the
compiler often cannot tell you at all: with `checkJs` off, a type written
for a function whose callers are still `.js` is checked against nothing.
This skill makes the claim falsifiable with a two-arm substitution proof at
a fixed commit: Arm A is the PR as written and must be silent, Arm B swaps
the hand-written type for the derived one and exercises it as the real call
site does. Each new diagnostic is a disagreement the hand-written type
concealed.
It also audits the second axis a migration can fail on — typing edits that
quietly change runtime behavior: stripped `| undefined`, deleted default
parameters, literals swapped for runtime enum lookups, and calls made
optional so a throw becomes a silent no-op.
Includes a runner script, a worked example (12 hand-written types, 5
confirmed divergences, 5 cleared falsifiers), and extension-specific notes.
Review-side companion to the "derive types from authoritative sources"
guideline in contributor-docs.
@MajorLift
MajorLift force-pushed the add-skill/typescript-type-proof branch from c5cd833 to b138a8d Compare July 30, 2026 13:21
…d add the false-negative catalog

The old name described what `tsc` already does. The skill's subject is the
complement: defects the compiler is structurally unable to report, because it
checks declarations for internal consistency and never for correspondence to the
source they restate.

Adds `references/false-negatives.md` covering the class that needs no PR to find —
unchecked index access, optional-vs-undefined, bivariant method parameters,
covariant arrays, excess-property checks that only fire on fresh literals,
structural erasure of domain types, `any` absorption, unverified `declare module`,
asserted external data, and the config-level gaps (`skipLibCheck`, `exclude`,
`checkJs`, and `noEmit` meaning a type error breaks a CI job rather than the
build). Verified against `metamask-extension` at `7fafda0` with a demonstration
file whose ten blocks are all wrong and on which `tsc` reports zero errors.
@MajorLift MajorLift changed the title feat(testing): add typescript-type-proof skill Add typescript-compiler-blindspots skill Jul 30, 2026
Six distributional claims across the skill and its references were asserted from
a single observed migration: "nearly every real finding", "most types have a
source", "the escape hatch usually exists to service the type", "often deletes
the reason a guard existed", "the single highest-frequency false negative in most
codebases", and "the one most likely to ship a real defect".

Where a count exists it is now cited (9 of 12 hand-written types had a source;
all five divergences fit the four shapes, stated as a small sample rather than a
partition). Where none exists the sentence is rewritten as an instruction to
check or as the mechanism itself, which is what each was doing anyway.
@MajorLift
MajorLift marked this pull request as draft July 30, 2026 14:03
A heading is read out of order, linked to directly, and shown in outlines and
search results, so it has no antecedent available: "any of this" and "use this in
a review" resolve only for a reader who arrived from the line above.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant