Skip to content

Add Foundry showmap relscore pipeline - #14

Merged
0xalpharush merged 8 commits into
tempoxyz:mainfrom
figtracer:codex/foundry-benchmark-roadmap
Jun 16, 2026
Merged

Add Foundry showmap relscore pipeline#14
0xalpharush merged 8 commits into
tempoxyz:mainfrom
figtracer:codex/foundry-benchmark-roadmap

Conversation

@figtracer

@figtracer figtracer commented Jun 16, 2026

Copy link
Copy Markdown

PR #11 is fragile because SCFuzzBench artifacts are uploaded per runner/fuzzer, while Foundry showmap output can be nested by approach, suite, test, and trial. This adds the missing normalization layer before scoring and runs normalized campaigns through the differential-coverage package for both relscore and relcov:

  • raw uploaded showmap/ trees are preserved for debugging
  • Foundry replay keeps normal FOUNDRY_TEST_ARGS such as RPC/thread flags, while script-owned showmap flags win during replay
  • showmap replay lets forge resolve corpus directories from Foundry config unless FOUNDRY_SHOWMAP_CORPUS_DIR/SCFUZZBENCH_CORPUS_DIR is explicitly set
  • analysis materializes canonical approach/trial.txt campaign directories under showmap_campaigns/
  • showmap_campaigns/combined/<approach>/<trial>.txt is the stable overall differential-coverage input
  • showmap_campaigns/by_test/<suite-test>/<approach>/<trial>.txt keeps per-test drill-down campaigns
  • differential_coverage_relscores.csv records relscore values from differential-coverage
  • differential_coverage_relcov.csv records pairwise relcov values from differential-coverage
  • showmap_campaign_manifest.json records included/skipped artifacts so missing or invalid showmap files are visible
  • filtered analysis applies the same fuzzer exclusions to showmap scoring as to event/throughput/progress CSVs

@figtracer figtracer closed this Jun 16, 2026
@figtracer figtracer reopened this Jun 16, 2026
@figtracer
figtracer force-pushed the codex/foundry-benchmark-roadmap branch from 2cf6b3c to 29dfd44 Compare June 16, 2026 09:25
@figtracer figtracer changed the title docs: add Foundry benchmarking roadmap Add Foundry showmap relscore pipeline Jun 16, 2026
@figtracer
figtracer marked this pull request as ready for review June 16, 2026 09:26
@figtracer
figtracer force-pushed the codex/foundry-benchmark-roadmap branch from 29dfd44 to f5230c5 Compare June 16, 2026 09:32
@figtracer
figtracer marked this pull request as draft June 16, 2026 09:35
@figtracer
figtracer force-pushed the codex/foundry-benchmark-roadmap branch 2 times, most recently from 93e579d to c847930 Compare June 16, 2026 09:42
@figtracer
figtracer force-pushed the codex/foundry-benchmark-roadmap branch from c847930 to 401056a Compare June 16, 2026 09:58
@socket-security

socket-security Bot commented Jun 16, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedpypi/​differential-coverage@​1.1.0100100100100100

View full report

@figtracer
figtracer marked this pull request as ready for review June 16, 2026 10:28
@figtracer
figtracer force-pushed the codex/foundry-benchmark-roadmap branch from 817c4ff to 082521f Compare June 16, 2026 10:33

@mablr mablr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check if we need to update benchmark-release.yml, to copy showmap_campaign_manifest.json and showmap_campaigns/ produced by analysis/analyze.py.

Comment thread fuzzers/foundry/run.sh
Comment thread analysis/analyze.py
Comment thread fuzzers/foundry/run.sh Outdated
Comment thread analysis/analyze.py
Comment thread analysis/analyze.py Outdated
Comment thread analysis/analyze.py Outdated
Comment thread .github/workflows/benchmark-release.yml

@mablr mablr left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

@0xalpharush
0xalpharush merged commit d675a88 into tempoxyz:main Jun 16, 2026
2 checks passed
@figtracer
figtracer deleted the codex/foundry-benchmark-roadmap branch June 17, 2026 12:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants