fix(ingestion): parse npm/pnpm lockfiles natively, drop snyk-nodejs-lockfile-parser#259
Merged
Merged
Conversation
…ockfile-parser `npm install -g @opencodehub/cli` emitted 5 deprecation warnings; 3 of them (lodash.clone, lodash.isequal, uuid@8) came from snyk-nodejs-lockfile-parser, a ~126-package transitive tree the CLI bundled solely to enumerate resolved packages from a lockfile. Its CJS graph also resisted tsup bundling (dynamic require of fs + @snyk/error-catalog-nodejs-public), so it could not be inlined. Replace it with a native parser. All we need from a lockfile is the flat set of resolved name@version pairs, and every shape already carries that verbatim: - npm v2/v3: the `packages` map keys each install by node_modules path + version - npm v1: the nested `dependencies` tree carries a version per node - pnpm 5/6/9: the `packages:`/`snapshots:` keys are name@version (or legacy /name/version), with peer suffixes stripped The license harvester already string-scanned the same raw lockfile, so the package list and its licenses now come from one pass over one source of truth. No YAML dependency added — pnpm keys are scanned the same way licenses already were. Public API is unchanged — this is an internal implementation swap, so it lands as a normal patch (no breaking change). Removes snyk-nodejs-lockfile-parser from both @opencodehub/ingestion and @opencodehub/cli. Verified end-to-end from the packed tarball: warnings drop 5 -> 2 (remaining glob/inflight are @sourcegraph/scip-python's, tracked upstream), and `codehub analyze` correctly indexes deps from both a v3 package-lock.json and a v9 pnpm-lock.yaml (incl. scoped names + peer-resolved versions). Docs: add a troubleshooting note that the residual glob/inflight warnings are cosmetic, and correct stale native-binding claims in the troubleshooting guide and CLAUDE.md — OpenCodeHub has zero native bindings (parsing is web-tree-sitter WASM, store is built-in node:sqlite, optional embedder is onnxruntime-web WASM; @duckdb/node-api was removed in ADR 0019, and there is no onnxruntime-node). Tests: +5 dep-parser fixtures (npm v1/v2/v3, pnpm v9/v6, missing-manifest); 638 ingestion tests pass, biome clean, docs build green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
theagenticguy
force-pushed
the
fix/replace-snyk-lockfile-parser
branch
from
71db7c0 to
0555388
Compare
Merged
theagenticguy
pushed a commit
that referenced
this pull request
Jun 28, 2026
🤖 Automated release via release-please --- <details><summary>root: 0.10.2</summary> ## [0.10.2](root-v0.10.1...root-v0.10.2) (2026-06-28) ### Bug Fixes * **ingestion:** parse npm/pnpm lockfiles natively, drop snyk-nodejs-lockfile-parser ([#259](#259)) ([738511b](738511b)) </details> <details><summary>cli: 0.10.2</summary> ## [0.10.2](cli-v0.10.1...cli-v0.10.2) (2026-06-28) ### Bug Fixes * **ingestion:** parse npm/pnpm lockfiles natively, drop snyk-nodejs-lockfile-parser ([#259](#259)) ([738511b](738511b)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
npm install -g @opencodehub/cliemitted 5 deprecation warnings. Three (lodash.clone,lodash.isequal,uuid@8) came fromsnyk-nodejs-lockfile-parser— a ~126-package transitive tree the CLI bundled solely to list resolved packages from a lockfile. This replaces it with a native parser and removes the dependency from both@opencodehub/ingestionand@opencodehub/cli.Result: warnings drop 5 → 2. (The remaining
glob/inflightcome from@sourcegraph/scip-pythonand need an upstream fix — tracked separately.)Why native parsing is the right call
All the dependency-ingestion phase needs from a lockfile is the flat set of resolved
name@versionpairs, and every lockfile shape already carries that verbatim:packagesmap — keyed bynode_modules/<name>path +versiondependenciestree —versionper nodepackages:/snapshots:keys —name@version(or legacy/name/version), peer suffixes strippedThe license harvester in this same file already string-scanned the raw lockfile, so the package list and its licenses now come from one pass over one source of truth. No YAML dependency added — pnpm keys are scanned exactly the way licenses already were.
This also sidesteps a real bundling failure: snyk's CJS graph does dynamic
require()offsand@snyk/error-catalog-nodejs-public, which broke when inlined via tsup (the alternative fix). Native parsing has no such hazard.Verification (end-to-end, from the packed tarball)
Built →
pnpm pack→ installed the tarball into a clean prefix → rancodehub analyze:snyk-nodejs-lockfile-parserabsent from the consumer tree; snyk symbols gone fromdist.package-lock.json:analyzeexit 0 — indexedleft-pad+@scope/util(scoped, with licenses).pnpm-lock.yaml:analyzeexit 0 — indexedleft-pad,@scope/util,react-dom(peer-suffix(react@18.2.0)correctly stripped).Gates:
typecheck✓ (CI-equivalent, excludes docs per ci.yml), 638 ingestion tests ✓ (incl. +5 new dep-parser fixtures: npm v1/v2/v3, pnpm v9/v6, missing-manifest),biome✓ (686 files), docs build ✓ (64 pages).Also in this PR
glob/inflightwarnings are not security issues (CI runs osv/grype/semgrep/npm-audit; overrides pin patched versions), and noting the lockfile-parser warnings are now gone.@duckdb/node-apireferences in the troubleshooting guide — DuckDB was removed in ADR 0019;onnxruntime-nodeis the only remaining native binding.Blast radius
Internal lockfile-parsing implementation only; no public API change. The dependency-ingestion phase output is unchanged (verified by the analyze runs + existing tests).
🤖 Generated with Claude Code