Skip to content

Reduce pull request performance checks - #99

Merged
SandroMaglione merged 3 commits into
mainfrom
codex/faster-performance-ci
Aug 13, 2026
Merged

Reduce pull request performance checks#99
SandroMaglione merged 3 commits into
mainfrom
codex/faster-performance-ci

Conversation

@SandroMaglione

@SandroMaglione SandroMaglione commented Aug 13, 2026

Copy link
Copy Markdown
Member

Summary

  • classify pull request changes so type performance, runtime performance, memory measurement, and examples run only when relevant
  • retain all 13 Effect Machine runtime scenarios and all 7 Effect Machine memory profiles while removing XState and Effect-runtime comparison baselines
  • keep five interleaved base/head processes and fail only large regressions beyond fixed and variability-adjusted thresholds
  • preserve stable required gate names when measurements or examples are skipped
  • stop duplicating CI on main and limit release runs to publishable changes

Changeset

  • Added for the package-metadata and benchmark dependency changes

Validation

  • pnpm check
  • pnpm perf:types
  • pnpm perf:runtime
  • clean-install pnpm perf:runtime -- --quick
  • workflow YAML parsing and focused classifier/regression tests

The focused full runtime suite completed locally in about 14 seconds for one
benchmark process while retaining every Effect Machine memory profile.

GitHub validation completed successfully:

  • CI: 1m28s, including all three examples selected by this infrastructure change
  • type performance: 1m34s
  • runtime and memory performance: 4m16s total, with 4m1s spent measuring five interleaved base/head processes
  • trusted type and runtime sticky-comment workflows: passed

The previous representative runtime workflow took 10m13s, including a 9m35s
measurement step. After this PR merges, runtime-performance can be added to
the required status checks using the stable gate introduced here.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Type performance

Measured with TypeScript 6.0.3 and skipLibCheck=true.

Scenario Base PR Difference
Effect only 55 55 0 (0.0%)
Import effect-machine 55 55 0 (0.0%)
Machine.defineStates (3 states) 2,965 2,965 0 (0.0%)
Machine.make (3 states, 2 events) 8,715 8,715 0 (0.0%)
machine.handle (3 states, 2 transitions) 23,492 23,492 0 (0.0%)
machine.handle (depth 24) 183,851 183,851 0 (0.0%)
machine.handle (wide depth 16) 214,189 214,189 0 (0.0%)
machine.handle (parallel/history/choice) 132,667 132,667 0 (0.0%)
machine.handle (4 successive calls) 128,674 128,674 0 (0.0%)
machine exact input/output/error/services 110,584 110,584 0 (0.0%)
execution adapter readiness 127,854 127,854 0 (0.0%)

Marginal instantiations are measured against the matching setup without that API call:

Scenario Base PR Difference
Import effect-machine 0 0 0
Machine.defineStates (3 states) 2,910 2,910 0 (0.0%)
Machine.make (3 states, 2 events) 5,742 5,742 0 (0.0%)
machine.handle (3 states, 2 transitions) 14,777 14,777 0 (0.0%)
machine.handle (depth 24) 171,577 171,577 0 (0.0%)
machine.handle (wide depth 16) 202,224 202,224 0 (0.0%)
machine.handle (parallel/history/choice) 114,331 114,331 0 (0.0%)
machine.handle (4 successive calls) 113,703 113,703 0 (0.0%)
machine exact input/output/error/services 100,670 100,670 0 (0.0%)
execution adapter readiness 102,092 102,092 0 (0.0%)
Check times (informational)
Scenario Base PR
Effect only 0.03s 0.03s
Import effect-machine 0.04s 0.03s
Machine.defineStates (3 states) 0.11s 0.10s
Machine.make (3 states, 2 events) 0.16s 0.16s
machine.handle (3 states, 2 transitions) 0.25s 0.23s
machine.handle (depth 24) 0.70s 0.69s
machine.handle (wide depth 16) 0.76s 0.76s
machine.handle (parallel/history/choice) 0.61s 0.63s
machine.handle (4 successive calls) 0.57s 0.58s
machine exact input/output/error/services 0.51s 0.51s
execution adapter readiness 0.61s 0.57s

Type instantiations are the comparison metric. Check time varies with runner load and is informational only.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Runtime performance

Median of 5 independent benchmark processes on AMD EPYC 9V74 80-Core Processor with Node v24.18.0.

Pull request baseline

Scenario Effect Machine
Plan counter transitions 129,627 transitions/s
Drain burst with terminal fence 518,323 increments/s
Drain burst with a change observer 478,528 increments/s
Lookup and send to one child 425,536 increments/s
Start and stop a machine 166,141 machines/s
Start and stop a parent with one child 44,045 families/s
Plan transitions through a compound state 115,946 transitions/s
Plan transitions through parallel regions 91,843 transitions/s
Drain burst through a compound state 447,429 events/s
Drain burst through two parallel regions 413,505 events/s
Drain a compound-state burst with a change observer 415,080 events/s

Effect runtime reference points

Scenario Effect Machine
Start and stop a raw generic process 27,215 processes/s
Start and stop a raw compiled process 70,817 processes/s
Memory profile Effect Machine
Idle machine 1.5 KiB
Raw generic managed process 12.7 KiB
Raw compiled process 2.8 KiB
Two independent idle machines 3.0 KiB
Idle parent with one child 4.8 KiB
Parent with observed child registry 9.0 KiB
Parent with observed invoked child snapshots 5.0 KiB

Effect Machine change from base

Metric Base Base variability PR PR variability Difference
Plan counter transitions 130,520 transitions/s 0.4% MAD 129,627 transitions/s 1.2% MAD -0.7%
Drain burst with terminal fence 525,392 increments/s 0.3% MAD 518,323 increments/s 1.7% MAD -1.3%
Drain burst with a change observer 482,627 increments/s 0.2% MAD 478,528 increments/s 1.4% MAD -0.8%
Lookup and send to one child 429,137 increments/s 0.2% MAD 425,536 increments/s 2.0% MAD -0.8%
Start and stop a machine 167,842 machines/s 0.3% MAD 166,141 machines/s 0.1% MAD -1.0%
Start and stop a parent with one child 45,080 families/s 0.5% MAD 44,045 families/s 0.4% MAD -2.3%
Plan transitions through a compound state 117,094 transitions/s 0.0% MAD 115,946 transitions/s 1.5% MAD -1.0%
Plan transitions through parallel regions 92,386 transitions/s 0.1% MAD 91,843 transitions/s 2.2% MAD -0.6%
Drain burst through a compound state 447,968 events/s 0.8% MAD 447,429 events/s 1.9% MAD -0.1%
Drain burst through two parallel regions 412,555 events/s 0.2% MAD 413,505 events/s 0.8% MAD +0.2%
Drain a compound-state burst with a change observer 414,212 events/s 1.0% MAD 415,080 events/s 2.3% MAD +0.2%
Idle machine heap per unit 1.5 KiB 0.1% MAD 1.5 KiB 0.1% MAD -0.2%
Raw generic managed process heap per unit 12.7 KiB 0.0% MAD 12.7 KiB 0.0% MAD +0.0%
Raw compiled process heap per unit 2.8 KiB 0.1% MAD 2.8 KiB 0.2% MAD -0.0%
Two independent idle machines heap per unit 3.0 KiB 0.0% MAD 3.0 KiB 0.1% MAD +0.1%
Idle parent with one child heap per unit 4.8 KiB 0.0% MAD 4.8 KiB 0.0% MAD +0.0%
Parent with observed child registry heap per unit 9.0 KiB 0.0% MAD 9.0 KiB 0.0% MAD +0.0%
Parent with observed invoked child snapshots heap per unit 5.0 KiB 0.1% MAD 5.0 KiB 0.1% MAD +0.2%

Effect runtime reference change from base

Metric Base Base variability PR PR variability Difference
Start and stop a raw generic process 27,259 processes/s 1.9% MAD 27,215 processes/s 1.4% MAD -0.2%
Start and stop a raw compiled process 71,018 processes/s 0.3% MAD 70,817 processes/s 0.6% MAD -0.3%
Versions and interpretation
  • Effect Machine: 0.6.0

Higher throughput is better; lower heap is better. Variability is the median absolute deviation across independent processes, relative to their median. Runtime measurements on shared GitHub-hosted hardware remain informational, so small differences should be confirmed across multiple workflow runs.

@SandroMaglione
SandroMaglione marked this pull request as ready for review August 13, 2026 08:35
@SandroMaglione
SandroMaglione merged commit efc4f2f into main Aug 13, 2026
13 checks passed
@SandroMaglione
SandroMaglione deleted the codex/faster-performance-ci branch August 13, 2026 08:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant