fix(ingestion): parse npm/pnpm lockfiles natively, drop snyk-nodejs-lockfile-parser

CLAUDE.md

@@ -92,8 +92,11 @@ storage bindings:** `@ladybugdb/core` AND `@duckdb/node-api` are both
 removed (ADR 0019 supersedes ADR 0016). The write-only Parquet embeddings
 sidecar (BOM item #7) was dropped with DuckDB — nothing ever read it back;
 embeddings live in the `embeddings` table in `store.sqlite`. The code-pack
-is now an 8-item BOM. (`onnxruntime-node`, the embedder, is the only
-remaining native dep — optional, lazy under `--embeddings`.)
+is now an 8-item BOM. **Zero native bindings, full stop:** the embedder is
+`onnxruntime-web` (prebuilt WASM, in `optionalDependencies`, lazy under
+`--embeddings`) — there is no `onnxruntime-node` and nothing compiles at
+install. Parsing is WASM (`web-tree-sitter`) and the store is the built-in
+`node:sqlite`, so the entire install is pure JS + WASM.
 
 Schema: one generic `nodes` table (typed base columns +
 `payload` JSON overflow for the 37 kind-specific shapes), one polymorphic

packages/cli/package.json

@@ -58,7 +58,6 @@
     "listr2": "10.2.1",
     "lru-cache": "11.5.1",
     "piscina": "5.2.0",
-    "snyk-nodejs-lockfile-parser": "2.8.1",
     "web-tree-sitter": "0.26.9",
     "write-file-atomic": "8.0.0",
     "yaml": "2.9.0",

packages/docs/src/content/docs/guides/troubleshooting.md

@@ -5,29 +5,32 @@ sidebar:
   order: 90
 ---
 
-## Native build failures
+## Install / `node-gyp` build failures
 
-Symptoms: `pnpm install` fails while building `@duckdb/node-api`. Error
-mentions `node-gyp`, `python`, a C/C++ compiler, or `Visual Studio
-Build Tools`.
+Symptoms: `npm install -g @opencodehub/cli` (or `pnpm install` from a
+checkout) fails with `node-gyp`, `python`, a C/C++ compiler, or `Visual
+Studio Build Tools` in the error.
 
-Fix:
+OpenCodeHub installs with **zero native bindings**, so it never compiles
+anything at install time. Every runtime component is pure JS or WASM:
+
+- Parsing is `web-tree-sitter` (WASM), with grammars vendored as `.wasm`
+  blobs — no native tree-sitter build.
+- The store is a single-file SQLite index via the built-in `node:sqlite`
+  (Node ≥ 24.15) — no native database binding.
+- The optional embedder is `onnxruntime-web` (prebuilt WASM), loaded lazily
+  only under `--embeddings` — no `onnxruntime-node`, no native ONNX build.
+
+So a `node-gyp` error almost always comes from an unrelated package in your
+own project's tree, not from OpenCodeHub. Confirm with:
 
-```bash title="probe the native toolchain"
+```bash title="probe the environment"
 codehub doctor
 ```
 
-`doctor` checks Node version, the platform's C/C++ toolchain, and
-whether each native module can load. Follow the remediation hints it
-prints.
-
-The parse runtime is `web-tree-sitter` (WASM) on every supported Node
-version, so a missing C/C++ toolchain does not break parsing. The only
-native bindings OpenCodeHub loads are `@duckdb/node-api` (temporal store)
-and `onnxruntime-node` (the local embedder) — both ship platform
-prebuilds, so a normal install does not compile anything. If a prebuild
-is missing for your platform, `codehub doctor` reports which module
-failed to load and prints the remediation steps.
+`doctor` checks the Node version and that each WASM/JS component loads. If
+it reports green but your install still fails, the failing module belongs to
+something else you are installing alongside the CLI.
 
 ## Stale index
 
@@ -62,23 +65,14 @@ for the exact shape.
 
 ## Windows quirks
 
-Parsing is WASM, so the parser needs no native toolchain on Windows. The
-native bindings (`@duckdb/node-api`, `onnxruntime-node`) ship `win32-x64`
-prebuilds, so a standard install pulls a binary rather than compiling.
-If a prebuild is unavailable and a module has to build from source, you
-need the Microsoft C++ Build Tools plus a matching Python for
-`node-gyp`. In practice the fastest fix is to run everything under WSL2 —
-WSL2 ships with a working toolchain out of the box and avoids path
-separator issues.
-
-If you must stay on native Windows and a source build is forced:
+OpenCodeHub has no native bindings, so a standard install never compiles
+on Windows — parsing is WASM (`web-tree-sitter`), the store is the built-in
+`node:sqlite`, and the optional embedder is `onnxruntime-web` (prebuilt
+WASM). There is no C/C++ toolchain requirement.
 
-1. Install Visual Studio Build Tools with the "Desktop development
-   with C++" workload.
-2. Install Python from the Microsoft Store (Python 3.12).
-3. `npm config set msvs_version 2022` and `npm config set python
-   python3.12`.
-4. Re-run `pnpm install --frozen-lockfile`.
+The remaining Windows friction is path-separator and shell quirks rather
+than builds. If you hit those, the smoothest environment is WSL2, which
+matches the POSIX paths the rest of the toolchain assumes.
 
 ## The index is missing a language I expected
 
@@ -88,6 +82,26 @@ language without a native toolchain. If the language is not listed,
 it is not yet registered — see
 [adding a language provider](/opencodehub/contributing/adding-a-language-provider/).
 
+## Deprecation warnings during `npm install -g @opencodehub/cli`
+
+Symptoms: `npm install -g @opencodehub/cli` prints `npm warn deprecated`
+lines for transitive packages such as `glob@7.2.3` and `inflight@1.0.6`.
+
+These are cosmetic. They are deprecation notices npm emits for indirect
+dependencies pulled in by a SCIP indexer the CLI ships
+(`@sourcegraph/scip-python` → `glob` → `inflight`). They are not security
+advisories: every published OpenCodeHub release passes osv-scanner, grype,
+semgrep, and npm-audit in CI, and pinned `overrides` hold transitive
+packages at patched versions. Nothing about the warnings affects install
+correctness or runtime behaviour, and there is no action for you to take.
+
+The lockfile-parser warnings (`lodash.clone`, `lodash.isequal`, `uuid@8`)
+that earlier releases also emitted are gone as of the native lockfile
+parser — the CLI no longer bundles a third-party resolver for dependency
+ingestion. The remaining `glob`/`inflight` pair originates inside the
+upstream indexer and is tracked for removal once that package updates its
+own dependencies.
+
 ## More help
 
 - `codehub doctor --verbose` dumps every probe the doctor runs.

packages/ingestion/package.json

@@ -55,7 +55,6 @@
     "graphology": "0.26.0",
     "graphology-dag": "0.4.1",
     "piscina": "5.2.0",
-    "snyk-nodejs-lockfile-parser": "2.8.1",
     "spdx-correct": "^3.2.0",
     "web-tree-sitter": "0.26.9",
     "write-file-atomic": "8.0.0"

packages/ingestion/src/pipeline/dep-parsers/npm.test.ts

@@ -76,6 +76,242 @@ describe("parseNpmDeps — package-lock.json (lockfileVersion 2)", () => {
   });
 });
 
+describe("parseNpmDeps — package-lock.json (lockfileVersion 3 + scoped + license)", () => {
+  let dir: string;
+
+  before(async () => {
+    dir = await mkdtemp(path.join(tmpdir(), "och-npm-lock3-"));
+    await writeFile(
+      path.join(dir, "package.json"),
+      JSON.stringify({ name: "fixture", version: "2.0.0" }),
+    );
+    // v3 lockfiles drop the legacy top-level `dependencies` mirror entirely
+    // and key everything under `packages`. Includes a scoped package and a
+    // `license` field to exercise the license join.
+    await writeFile(
+      path.join(dir, "package-lock.json"),
+      JSON.stringify({
+        name: "fixture",
+        version: "2.0.0",
+        lockfileVersion: 3,
+        requires: true,
+        packages: {
+          "": { name: "fixture", version: "2.0.0" },
+          "node_modules/left-pad": { version: "1.3.0", license: "WTFPL" },
+          "node_modules/@scope/util": { version: "4.5.6", license: "MIT" },
+          // Nested transitive (deduped npm layout) — still a resolved pkg.
+          "node_modules/left-pad/node_modules/semver": { version: "7.6.0" },
+        },
+      }),
+    );
+  });
+
+  after(async () => {
+    await rm(dir, { recursive: true, force: true });
+  });
+
+  it("emits every resolved package incl. scoped, nested, and license", async () => {
+    const warnings: string[] = [];
+    const out = await parseNpmDeps({
+      absPath: path.join(dir, "package-lock.json"),
+      relPath: "package-lock.json",
+      repoRoot: dir,
+      onWarn: (m) => warnings.push(m),
+    });
+    assert.equal(warnings.length, 0, `unexpected warnings: ${warnings.join("\n")}`);
+    const byName = new Map(out.map((d) => [d.name, d]));
+    assert.equal(byName.get("left-pad")?.version, "1.3.0");
+    assert.equal(byName.get("left-pad")?.license, "WTFPL");
+    assert.equal(byName.get("@scope/util")?.version, "4.5.6");
+    assert.equal(byName.get("@scope/util")?.license, "MIT");
+    // nested transitive captured by node_modules path tail
+    assert.equal(byName.get("semver")?.version, "7.6.0");
+    // root project itself must NOT appear as a dependency
+    assert.equal(byName.has("fixture"), false);
+  });
+});
+
+describe("parseNpmDeps — package-lock.json (legacy lockfileVersion 1)", () => {
+  let dir: string;
+
+  before(async () => {
+    dir = await mkdtemp(path.join(tmpdir(), "och-npm-lock1-"));
+    await writeFile(
+      path.join(dir, "package.json"),
+      JSON.stringify({ name: "fixture", version: "1.0.0" }),
+    );
+    // v1 has no `packages` map — only the nested `dependencies` tree.
+    await writeFile(
+      path.join(dir, "package-lock.json"),
+      JSON.stringify({
+        name: "fixture",
+        version: "1.0.0",
+        lockfileVersion: 1,
+        requires: true,
+        dependencies: {
+          "left-pad": { version: "1.3.0" },
+          minimist: {
+            version: "1.2.8",
+            dependencies: { "nested-dep": { version: "0.0.1" } },
+          },
+        },
+      }),
+    );
+  });
+
+  after(async () => {
+    await rm(dir, { recursive: true, force: true });
+  });
+
+  it("walks the nested dependencies tree", async () => {
+    const out = await parseNpmDeps({
+      absPath: path.join(dir, "package-lock.json"),
+      relPath: "package-lock.json",
+      repoRoot: dir,
+      onWarn: () => {},
+    });
+    const byName = new Map(out.map((d) => [d.name, d]));
+    assert.equal(byName.get("left-pad")?.version, "1.3.0");
+    assert.equal(byName.get("minimist")?.version, "1.2.8");
+    assert.equal(byName.get("nested-dep")?.version, "0.0.1");
+  });
+});
+
+describe("parseNpmDeps — pnpm-lock.yaml (v9 modern keys)", () => {
+  let dir: string;
+
+  before(async () => {
+    dir = await mkdtemp(path.join(tmpdir(), "och-pnpm9-"));
+    await writeFile(
+      path.join(dir, "package.json"),
+      JSON.stringify({ name: "fixture", version: "1.0.0" }),
+    );
+    // v9 keys are `name@version` / `@scope/name@version`, optionally with a
+    // `(peerHash)` suffix under both `packages:` and `snapshots:`.
+    await writeFile(
+      path.join(dir, "pnpm-lock.yaml"),
+      [
+        "lockfileVersion: '9.0'",
+        "",
+        "packages:",
+        "",
+        "  left-pad@1.3.0:",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+        "  '@scope/util@4.5.6':",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+        "  react-dom@18.2.0(react@18.2.0):",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+        "snapshots:",
+        "",
+        "  left-pad@1.3.0: {}",
+        "",
+        "  '@scope/util@4.5.6': {}",
+        "",
+      ].join("\n"),
+    );
+  });
+
+  after(async () => {
+    await rm(dir, { recursive: true, force: true });
+  });
+
+  it("parses modern keys incl. scoped and peer-suffixed versions", async () => {
+    const warnings: string[] = [];
+    const out = await parseNpmDeps({
+      absPath: path.join(dir, "pnpm-lock.yaml"),
+      relPath: "pnpm-lock.yaml",
+      repoRoot: dir,
+      onWarn: (m) => warnings.push(m),
+    });
+    assert.equal(warnings.length, 0, `unexpected warnings: ${warnings.join("\n")}`);
+    const byName = new Map(out.map((d) => [d.name, d]));
+    assert.equal(byName.get("left-pad")?.version, "1.3.0");
+    assert.equal(byName.get("@scope/util")?.version, "4.5.6");
+    // peer suffix stripped to the bare version
+    assert.equal(byName.get("react-dom")?.version, "18.2.0");
+  });
+});
+
+describe("parseNpmDeps — pnpm-lock.yaml (legacy v6 slash keys)", () => {
+  let dir: string;
+
+  before(async () => {
+    dir = await mkdtemp(path.join(tmpdir(), "och-pnpm6-"));
+    await writeFile(
+      path.join(dir, "package.json"),
+      JSON.stringify({ name: "fixture", version: "1.0.0" }),
+    );
+    // v5/v6 keys are `/name/version` / `/@scope/name/version`, with optional
+    // `_peer` or `(peer)` suffix.
+    await writeFile(
+      path.join(dir, "pnpm-lock.yaml"),
+      [
+        "lockfileVersion: '6.0'",
+        "",
+        "packages:",
+        "",
+        "  /left-pad/1.3.0:",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+        "  /@scope/util/4.5.6:",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+        "  /react-dom/18.2.0_react@18.2.0:",
+        "    resolution: {integrity: sha512-fake==}",
+        "",
+      ].join("\n"),
+    );
+  });
+
+  after(async () => {
+    await rm(dir, { recursive: true, force: true });
+  });
+
+  it("parses legacy slash keys incl. scoped and peer suffix", async () => {
+    const out = await parseNpmDeps({
+      absPath: path.join(dir, "pnpm-lock.yaml"),
+      relPath: "pnpm-lock.yaml",
+      repoRoot: dir,
+      onWarn: () => {},
+    });
+    const byName = new Map(out.map((d) => [d.name, d]));
+    assert.equal(byName.get("left-pad")?.version, "1.3.0");
+    assert.equal(byName.get("@scope/util")?.version, "4.5.6");
+    assert.equal(byName.get("react-dom")?.version, "18.2.0");
+  });
+});
+
+describe("parseNpmDeps — lockfile without sibling package.json", () => {
+  let dir: string;
+  before(async () => {
+    dir = await mkdtemp(path.join(tmpdir(), "och-nolock-"));
+    await writeFile(
+      path.join(dir, "package-lock.json"),
+      JSON.stringify({ name: "x", version: "1.0.0", lockfileVersion: 3, packages: {} }),
+    );
+  });
+  after(async () => {
+    await rm(dir, { recursive: true, force: true });
+  });
+  it("warns about the missing manifest and returns []", async () => {
+    const warnings: string[] = [];
+    const out = await parseNpmDeps({
+      absPath: path.join(dir, "package-lock.json"),
+      relPath: "package-lock.json",
+      repoRoot: dir,
+      onWarn: (m) => warnings.push(m),
+    });
+    assert.deepEqual([...out], []);
+    assert.ok(
+      warnings.some((w) => w.includes("lacks sibling package.json")),
+      `expected sibling-manifest warning, got: ${warnings.join("\n")}`,
+    );
+  });
+});
+
 describe("parseNpmDeps — bare package.json fallback", () => {
   let dir: string;