Skip to content

feat(bench): seed changesets from the fixture's commit history#176

Merged
BryanFRD merged 1 commit into
mainfrom
feat/seed-changesets
Jul 16, 2026
Merged

feat(bench): seed changesets from the fixture's commit history#176
BryanFRD merged 1 commit into
mainfrom
feat/seed-changesets

Conversation

@BryanFRD

Copy link
Copy Markdown
Contributor

Closes FerrLabs/Fixtures#141 — but not the way that issue predicted, and the headline result is negative. Read the last section before merging.

changesets status runs against fixtures with no .changeset/*.md in six of seven cases, so it prints "NO packages to be bumped" and exits. We publish that as changesets: 733 ms on the perf page. We are timing a no-op and presenting it as a release-planning benchmark.

This writes the changesets a maintainer would have written for the same history: one per package with a releasable commit since the release tag, at the level FerrFlow derives from the same commits (breaking > feat > fix/perf/refactor, mirroring src/conventional_commits.rs). On mono-large that is 200 changesets — 189 major, 11 minor — and changesets status now reports 200 packages to bump instead of none.

It does not change the numbers, and that is the finding

I A/B'd it on the real mono-large fixture, 7 runs each, same machine:

changesets status, no changesets ("NO packages to be bumped"):  1446 ms
changesets status, 200 seeded changesets (200 reported):        1411 ms
delta:                                                            -35 ms

Noise. Reading 200 changeset files is free. Fixtures#141's premise — mine — was wrong: giving changesets real input does not make the comparison fair, because changesets' work is intrinsically ~160 ms regardless. It never reads the 10k commits and never will; that is its design. The cost lives with the developer who wrote each changeset by hand at commit time, and no benchmark of the CLI can see it.

So this does not fix the perf page. changesets will still "win" 200 x 10k against a tool that walked 10k commits.

Why it is still worth merging

It fixes what the number means. 733 ms today is changesets doing nothing. 733 ms after this is changesets planning a real 200-package release — of which ~595 ms is Node booting and ~160 ms is reading changesets a human already wrote. Same figure, but now it is a defensible sentence instead of a misleading one.

What actually needs deciding

changesets does not derive releases from git history, so "check latency on a 10k-commit repo" is a question it does not answer. It arguably does not belong in the latency head-to-head at all (install footprint is a fair comparison and it already appears there). That cuts both ways and is the point: dropping it also removes every row where it makes us look 30x better on a fixture where it did nothing. Not my call — flagging it.

Shape

seed-changesets.sh, invoked from run.sh next to the existing changesets npm install, no-op when a fixture ships changesets by hand (single seeds one on purpose) or has no changesets config. The package set comes from the workspace, not from commit scopes: the single fixture scopes commits by area (core, api, cli), and naming a package changesets can't resolve is a hard error. One awk pass rather than a shell loop — mono-large is 10k commits and the bash-regex version took minutes.

9 bats tests: level derivation, breaking > feat > fix precedence, non-releasable types producing nothing, the single-package scope trap, unknown scopes, both no-op paths, and the empty range.

@BryanFRD BryanFRD merged commit c305331 into main Jul 16, 2026
15 checks passed
@BryanFRD BryanFRD deleted the feat/seed-changesets branch July 16, 2026 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fixtures contain no changeset files, so the changesets benchmark measures nothing

1 participant