Skip to content

feat(security): add supply-chain-audit and lavamoat-policy-diligence skills - #83

Open
MajorLift wants to merge 4 commits into
mainfrom
jongsun/add/security-domain
Open

feat(security): add supply-chain-audit and lavamoat-policy-diligence skills#83
MajorLift wants to merge 4 commits into
mainfrom
jongsun/add/security-domain

Conversation

@MajorLift

@MajorLift MajorLift commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Adds a security domain with two skills that compose into one question: is this dependency change safe to take?

supply-chain-audit — the breadth pass

Runs every detector that answers a different part of that question, and treats disposition as the deliverable rather than detection (detection belongs to the tools):

  • Socket Security — malicious/anomalous package behavior, install scripts, new maintainers
  • yarn npm audit and advisories — known vulnerabilities
  • Lockfile and manifest diffs — what actually changed, including transitive and resolution swaps
  • LavaMoat policy grants — delegated to the second skill

The part that earns it a place: the fronts no upstream scanner sees, because they are things your own repo does to dependencies after they arrive

Front Why a scanner misses it
patches/ applied at install modifies dependency source after the registry copy Socket scanned
resolutions forcing or stubbing versions the resolved tree differs from what the manifest declares
npmAuditIgnoreAdvisories a suppression list is invisible to the audit it suppresses
CI actions on mutable tags @v4 is a moving target; the SHA that runs tomorrow is not today's
yarn plugins execute at install, inside your trust boundary

The falsifier is a lane whose finding is unaccounted for — a flagged package, an unresolved advisory, or a grant with no call site.

lavamoat-policy-diligence — the depth pass

Capability containment, delegated from the lane above.

The framing that makes it useful: a LavaMoat policy is generated from a real run, so every grant has a call site by construction. That makes "each addition is justified" a tautology, and therefore not a deliverable. What is a deliverable is each grant's gate — a config flag nobody sets, an API nobody calls, a branch our payloads never take, an error-only path — read at the installed version, sorted into removable / removable-at-a-cost / load-bearing, with the removal test (drop it, rebuild, run e2e) proposed for the policy owners to run.

Ships scripts/policy-audit.py, which turns a base/head policy pair into a per-grant worklist.

Neither skill renders an accept/reject verdict. That call is the reviewer's, and the disposition belongs to the people who own the dependency.

Files

domains/security/skills/{supply-chain-audit,lavamoat-policy-diligence}/. Both experimental; no repos/ overlays. Commits GPG-signed.

Showcase — what lavamoat-policy-diligence produces

extension#42867 — @sentry/browser 8.33.1 → 10.38.0. The bump added grants across the @sentry/* subtree. A reviewer asked what two of them were for. Each was answered with the upstream line, pinned to the installed version:

Both sit on unconditional paths — the exception path and module scope — and under scuttling the read itself throws unless excepted. So both are load-bearing with no gate to close. That is the useful output: not "each grant has a reason" but "neither is removable, and here is the unconditional path that makes it so."

The counter-example is the reason the skill exists. A first pass on extension#45024 led with "11 additions, 11 reasons, each resolving to a line." That is tautological: the policy is generated from a real run, so every grant has a call site by construction and the count was guaranteed before anyone looked. Reading for gates — a config flag nobody sets, an API nobody calls, a branch our payloads never take — is what separates a removable grant from a load-bearing one.

Showcase — what supply-chain-audit produces

extension#44865 — clear the postcss advisories, which started as a yarn npm audit report of newly blocking advisories on main.

Detection was the easy half: four in-range advisories, two of them published against postcss 8 — GHSA-6g55-p6wh-862q (arbitrary file read via attacker-controlled sourceMappingURL, ≤ 8.5.11) and GHSA-r28c-9q8g-f849 (path traversal in source-map auto-loading, ≤ 8.5.17). What the audit could not say is what to do about eleven postcss-7 paths that no upstream bump reaches.

The disposition is the deliverable:

  • Find the owner rather than pinning the symptom. stylelint 13.6.1 owned 8 of the 11 postcss-7 paths; upgrading it to 17.14.1 removed them at the source.
  • Patch what cannot be upgraded. The two survivors — 7.0.36 under resolve-url-loader, 7.0.39 under gulp-sourcemaps — carry backports of all four advisory fixes in .yarn/patches, scoped so the v8 tree stays on 8.5.21. This is the lane no upstream scanner sees: Socket and audit examine a dependency as published, not as your repo patches it at install.
  • Reject a fix that would have made it worse. A competing PR proposed a blanket resolutions.postcss. Falsifier: does any postcss-7 consumer declare postcss as a peer rather than a hard dependency? All five declare a hard ^7postcss-scss@2.1.1, postcss-less@3.1.4, sugarss@2.0.0, postcss-safe-parser@4.0.2, autoprefixer@9.8.x — so a blanket resolution would force them onto v8 and break them. That PR was closed.
  • Prove no resolutions is needed at all. Falsifier: any ^8 descriptor resolving to a second copy. All nine map to one postcss@npm:8.5.21 entry in the committed lockfile.
  • Leave the remainder tracked, not implied. The 8.5.23 floor and the last two postcss-7 copies are an open follow-up rather than a sentence in a merged PR.

Every claim carries the observation that would have refuted it. That is the shape the skill asks for: the tools produce the worklist, and the audit produces the disposition and the reason.

Notes for reviewers


Validation runs

Trial runs of this PR's skills against merged metamask-extension PRs nobody flagged. Every claim was re-verified against the real diff before posting. Clean results are included on purpose — a skill that only ever reports problems cannot be calibrated.

PR Skill Verdict Finding
#42867 supply-chain-audit Gap @sentry/browser gains fetch, importScripts, WebAssembly
#44187 lavamoat-policy-diligence Mixed SW enters LavaMoat; hand-written override grants write
#44678 supply-chain-audit Gap sass-embedded bump grants worker_threads + fs
#44862 supply-chain-audit Note title names tar; the policy delta comes from streamx

Each comment carries a trial-run disclaimer and links back here for feedback.

…oat-policy-diligence

Two composing skills for "is this dependency change safe to take".

`supply-chain-audit` is the breadth pass: Socket findings, `yarn npm audit`
advisories, lockfile and manifest diffs, and the fronts no upstream scanner sees
because they are things the repo does to its dependencies afterwards — yarn
patches that modify dependency source at install, `resolutions` that force or
stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding
mutable tags, and yarn plugins that execute at install.

`lavamoat-policy-diligence` is the depth pass it delegates capability
containment to. Because a LavaMoat policy is generated from a real run, every
grant has a call site by construction — so "each addition is justified" is a
tautology, not a finding. It instead reads each grant's use at the installed
version to find its gate, and sorts into removable / removable-at-a-cost /
load-bearing.

Neither renders an accept/reject verdict; disposition belongs to the people who
own the dependency.

Adds a CODEOWNERS entry for the new domain, defaulted to the platform teams.
@MajorLift MajorLift changed the title feat(security): add security domain — supply-chain-audit and lavamoat-policy-diligence feat(security): add supply-chain-audit and lavamoat-policy-diligence skills Jul 30, 2026
@MajorLift MajorLift changed the title feat(security): add supply-chain-audit and lavamoat-policy-diligence skills feat(security): add supply-chain-audit and lavamoat-policy-diligence skills Jul 30, 2026
@MajorLift
MajorLift marked this pull request as draft July 30, 2026 14:03
@MajorLift
MajorLift marked this pull request as ready for review July 30, 2026 18:15
@MajorLift

MajorLift commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Context budget

What this PR costs an agent, measured from an install rather than read from the diff. Three tiers, and only the first is unavoidable.

Skill Frontmatter Selected + refs & knowledge
lavamoat-policy-diligence 1,177 chars ~4,126 tok ~4,673 tok
supply-chain-audit 1,314 chars ~2,953 tok ~2,953 tok

Frontmatter is the only tier paid unconditionally — every agent loads it on every run once the skill is installed, used or not, because it is what the agent reads to decide relevance. The 28 skills across the eleven open skill PRs sit at a median of ~1,716 tokens selected and ~1,860 with references followed. All are within the 1,536-character description budget.

Selected is paid only when the agent picks the skill. + refs & knowledge is the ceiling if every bundled reference is then read; it is a worst case, not an expectation.

Method

tools/install --repo metamask-extension --maturity experimental against this branch at d9c6231a1, measured per installed skill directory. Repo overlays are merged into the emitted SKILL.md, so they land in the selected tier rather than being missed by a source-byte count. Token figures are bytes/4 — a proxy for scale, not accounting.

These figures are pinned to the commit above and drift on every push; #96 tracks automating them.

`pr-validate` was renamed to `evidence`; both skills still named the old one, in
a section heading, prose, and a `## Related` entry. `supply-chain-audit` now also
links its evidence category in the catalog rather than naming it bare.
…rants

Adds `--override`: overrides are where containment is widened by a person rather
than observed by the toolchain, and the decision persists across regenerations,
so it outlives the reason for it. On the extension's mv3/main policy: 42
tightened, 83 persisting, 9 narrowable, 2 write.

Three things came from reading lavamoat-core/src/mergePolicy.js rather than
inferring them, each of which had been wrong:

- The effective policy is mergePolicy(generated, override). The two files are
  DESIGNED not to align — the generated one is regenerated on dependency updates
  while the override persists — so an override entry absent from the generated
  policy is the normal case. Calling that "never observed" was alarmist and
  wrong; 83 of 87 entries are in that state by construction.
- `validateHierarchy` throws when both `X` and `X.y` are present, so the first
  version of the narrowing suggestion would have produced a policy that fails to
  build. The documented form denies the parent: `"X": false, "X.y": true`.
- Escalation flagged every global in any package matching a name hint, labelling
  `Array` and `Object` as "critical class". Intrinsics are now excluded and the
  reason string is true of the row it appears on — an escalation list that is
  mostly noise trains its reader to skip it.

Critical grants and write access get RAISE WITH A HUMAN and no verdict.
Correctness there depends on intent and threat model, neither of which is in the
policy files, and an audit that silently resolves them has substituted a guess
for the thing it was asked to check.
A single escalation list mixes two different questions. A capability granted to a
package the base policy did not contain arrives because the package arrived — the
question there is whether the package belongs in the bundle. A capability newly
granted to a package already contained is somebody's decision about that
capability. Mixed together on one mv3 policy, 14 of the 24 rows were the first
kind, and a list that is mostly not actionable teaches its reader to skim.

Write access is exempt from the row cap: it is the smallest and highest-signal
category, and truncating it hides the row that most needed a reader.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant