Skip to content

javascript.md

Latest commit

 

History

History
91 lines (74 loc) · 4.77 KB

javascript.md

File metadata and controls

91 lines (74 loc) · 4.77 KB
Raw

JavaScript / TypeScript (npm)

The npm slice of the Value pipeline: how npm download and dependency data becomes a download-weighted PageRank and an A/B/C value class for every JavaScript/TypeScript package. This page covers the pipeline assembly; for raw-fetch mechanics (endpoints, rate limits, fetch scripts) see the source reference sources/npm.md.

Sources & data collected

Source Data collected Raw file (data/sources/npm/)
npm downloads API per-package annual downloads (2021–2025) raw/downloads.csv
npm downloads API ecosystem-wide annual totals (the 95% denominator) raw/npm-stats.csv
npm registry declared runtime dependencies (package → dep) raw/dependencies.csv
nice-registry package → repo_url mapping (~2M packages) nice-registry/packages.csv

No authentication required (an optional NPM_TOKEN raises the registry rate limit for dependency lookups). npm publishes no fixed rate limit for the downloads API; sustained ~1 req/s is the measured-safe rate, and the fetcher enforces it with a global rate limiter.

Value pipeline

npm data flows through the shared Value mechanics (full description in value.md):

  1. Top packages — sort by avg annual downloads, keep packages covering 95% of the ecosystem-wide total (from npm-stats.csv).
  2. Dependency tree — follow transitive runtime deps from the top set, fetching any missing deps from the registry.
  3. package → repo — match every dep-tree package against nice-registry.
  4. PageRank — download-weighted personalized PageRank (α = 0.85) over the directed dep graph (A → B means A depends on B).
  5. Value class — sort by PageRank desc; cumulative-share cutoffs assign A (≤75%) / B (≤95%) / C (rest).

Orchestrated by src.value.npm_pipeline (fetch-data → fetch-stats → fetch-repos → process). Metric lineage (← = data source, […] = period):

JavaScript / TypeScript (npm)
├── downloads_2021..2025   ← api.npmjs.org/downloads             [2021–2025]
├── avg_downloads          ← derived (mean over populated years) [2021–2025]
├── avg_downloads_share    ← derived (pkg / ecosystem total)     [2021–2025]
├── top                    ← derived (95% cum-download cutoff)   [2021–2025]
├── dep edges (package→dep)← registry.npmjs.org                  [most recent]
├── pagerank               ← derived (DL-weighted PR, α=0.85)    [2021–2025]
├── value_class            ← derived (A/B/C, cum-PR share)       [2021–2025]
└── package→repo           ← nice-registry                       [most recent]

Where it's used downstream

  • Value — each package's value_class is grouped by repo into data/value/value.csv as the class_npm column; the strongest class across ecosystems becomes class.
  • Risk — class-A npm repos enter src.risk.run_risk_pipeline (scope set by risk_input.value_classes in src/settings.json).
  • Eligibility — the same class-A repos (archived included) enter the automated Eligibility stage (src.eligibility.run_eligibility_pipeline), also keyed off github_repo. The per-ecosystem signals feed it: fetch_licenses.py fills the license column of results.csv (the registry-first input to the stage's license check), and check_eol.py → data/sources/npm/eol.csv produces advisory package-level EOL signals that inform the manual eol override in data/eligibility/overrides.csv.

Outputs

results.csv (data/sources/npm/) — one row per dep-tree package:

Column Description
package Package name
github_repo owner/repo slug
git, eco_guess Canonical git URL + identity provenance (eco / native / override), rewritten by the value rollup's ecosyste.ms authority pass (src.value.apply_ecosystems_authority)
avg_downloads, 2021–2025 Downloads
top True if in the 95% cumulative set
pagerank Download-weighted PageRank score
value_class A/B/C
repo_id, mirror_url Stable numeric GitHub repo id; upstream URL when the GitHub repo is a mirror
license SPDX license (filled by fetch_licenses.py)

npm funnel & classes

See the preview stats sheet → Value for the npm funnel counts (top packages → dep tree → results → repo coverage) and class distribution.

npm has the cleanest upstream identity (highest GitHub-repo coverage) of the four ecosystems, so essentially all load-bearing npm packages reach Risk and Eligibility.