feat(bench): seed changesets from the fixture's commit history#176
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes FerrLabs/Fixtures#141 — but not the way that issue predicted, and the headline result is negative. Read the last section before merging.
changesets statusruns against fixtures with no.changeset/*.mdin six of seven cases, so it prints "NO packages to be bumped" and exits. We publish that aschangesets: 733 mson the perf page. We are timing a no-op and presenting it as a release-planning benchmark.This writes the changesets a maintainer would have written for the same history: one per package with a releasable commit since the release tag, at the level FerrFlow derives from the same commits (breaking > feat > fix/perf/refactor, mirroring
src/conventional_commits.rs). Onmono-largethat is 200 changesets — 189 major, 11 minor — andchangesets statusnow reports 200 packages to bump instead of none.It does not change the numbers, and that is the finding
I A/B'd it on the real
mono-largefixture, 7 runs each, same machine:Noise. Reading 200 changeset files is free. Fixtures#141's premise — mine — was wrong: giving changesets real input does not make the comparison fair, because changesets' work is intrinsically ~160 ms regardless. It never reads the 10k commits and never will; that is its design. The cost lives with the developer who wrote each changeset by hand at commit time, and no benchmark of the CLI can see it.
So this does not fix the perf page.
changesetswill still "win"200 x 10kagainst a tool that walked 10k commits.Why it is still worth merging
It fixes what the number means.
733 mstoday is changesets doing nothing.733 msafter this is changesets planning a real 200-package release — of which ~595 ms is Node booting and ~160 ms is reading changesets a human already wrote. Same figure, but now it is a defensible sentence instead of a misleading one.What actually needs deciding
changesets does not derive releases from git history, so "check latency on a 10k-commit repo" is a question it does not answer. It arguably does not belong in the latency head-to-head at all (install footprint is a fair comparison and it already appears there). That cuts both ways and is the point: dropping it also removes every row where it makes us look 30x better on a fixture where it did nothing. Not my call — flagging it.
Shape
seed-changesets.sh, invoked fromrun.shnext to the existing changesetsnpm install, no-op when a fixture ships changesets by hand (singleseeds one on purpose) or has no changesets config. The package set comes from the workspace, not from commit scopes: thesinglefixture scopes commits by area (core,api,cli), and naming a package changesets can't resolve is a hard error. Oneawkpass rather than a shell loop —mono-largeis 10k commits and the bash-regex version took minutes.9 bats tests: level derivation, breaking > feat > fix precedence, non-releasable types producing nothing, the single-package scope trap, unknown scopes, both no-op paths, and the empty range.