Skip to content

feat(ci): benchmark a startup floor alongside every shard#678

Merged
BryanFRD merged 1 commit into
mainfrom
feat/bench-floor-fixture
Jul 15, 2026
Merged

feat(ci): benchmark a startup floor alongside every shard#678
BryanFRD merged 1 commit into
mainfrom
feat/bench-floor-fixture

Conversation

@BryanFRD

Copy link
Copy Markdown
Contributor

Closes #677. 2/3 — picks up FerrLabs/Benchmarks#173; the chart is FerrLabs/FerrFlow-Cloud.

Adds benchmarks/fixtures/definitions/floor.json: 1 package, 1 commit since the last tag, with the same tool_configs as single so every tool runs its normal command down its normal code path. That median is each tool's fixed cost of being invoked at all — process spawn, runtime boot, module loading, config resolution — and the action subtracts it to publish work_ms next to median_ms.

The floor rides along in every shard rather than being a shard of its own. Startup is a property of the runner, and shards land on different machines — subtracting one shard's floor from another's total would measure the hardware gap instead of the work. It costs one extra tiny fixture per shard (1 commit), which is noise next to mono-large.

--version is not the floor, though it looks like one: it skips the lazy requires the real command pulls in, undercounting startup by ~286 ms for commit-and-tag-version and billing the difference as their work. That error flatters us, which is why the floor runs the real command.

median_ms stays the headline and is untouched — startup is real time the user waits. Same rule as --jobs 1 vs ferrflow_parallel and cold vs ferrflow_cached: our nuances go in separate stats, never the score.

Expect the first run after this to add startup_ms / work_ms to every cell. The site renders them in the third PR; until then the payload just carries them.

@BryanFRD BryanFRD enabled auto-merge (squash) July 15, 2026 18:09
@BryanFRD BryanFRD merged commit 2e6fc20 into main Jul 15, 2026
38 of 39 checks passed
@BryanFRD BryanFRD deleted the feat/bench-floor-fixture branch July 15, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark bars can't distinguish a slow tool from a slow runtime boot

1 participant