Add Foundry showmap relscore pipeline - #14
Merged
0xalpharush merged 8 commits intoJun 16, 2026
Conversation
figtracer
force-pushed
the
codex/foundry-benchmark-roadmap
branch
from
June 16, 2026 09:25
2cf6b3c to
29dfd44
Compare
figtracer
marked this pull request as ready for review
June 16, 2026 09:26
figtracer
force-pushed
the
codex/foundry-benchmark-roadmap
branch
from
June 16, 2026 09:32
29dfd44 to
f5230c5
Compare
figtracer
marked this pull request as draft
June 16, 2026 09:35
figtracer
force-pushed
the
codex/foundry-benchmark-roadmap
branch
2 times, most recently
from
June 16, 2026 09:42
93e579d to
c847930
Compare
figtracer
force-pushed
the
codex/foundry-benchmark-roadmap
branch
from
June 16, 2026 09:58
c847930 to
401056a
Compare
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
figtracer
marked this pull request as ready for review
June 16, 2026 10:28
Amp-Thread-ID: https://ampcode.com/threads/T-019ecfdf-afb6-714c-9bd9-9b64a2f1caf0 Co-authored-by: Amp <amp@ampcode.com>
figtracer
force-pushed
the
codex/foundry-benchmark-roadmap
branch
from
June 16, 2026 10:33
817c4ff to
082521f
Compare
mablr
reviewed
Jun 16, 2026
mablr
left a comment
There was a problem hiding this comment.
Please check if we need to update benchmark-release.yml, to copy showmap_campaign_manifest.json and showmap_campaigns/ produced by analysis/analyze.py.
mablr
reviewed
Jun 16, 2026
mablr
reviewed
Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR #11 is fragile because SCFuzzBench artifacts are uploaded per runner/fuzzer, while Foundry showmap output can be nested by approach, suite, test, and trial. This adds the missing normalization layer before scoring and runs normalized campaigns through the
differential-coveragepackage for bothrelscoreandrelcov:showmap/trees are preserved for debuggingFOUNDRY_TEST_ARGSsuch as RPC/thread flags, while script-owned showmap flags win during replayforgeresolve corpus directories from Foundry config unlessFOUNDRY_SHOWMAP_CORPUS_DIR/SCFUZZBENCH_CORPUS_DIRis explicitly setapproach/trial.txtcampaign directories undershowmap_campaigns/showmap_campaigns/combined/<approach>/<trial>.txtis the stable overall differential-coverage inputshowmap_campaigns/by_test/<suite-test>/<approach>/<trial>.txtkeeps per-test drill-down campaignsdifferential_coverage_relscores.csvrecords relscore values fromdifferential-coveragedifferential_coverage_relcov.csvrecords pairwise relcov values fromdifferential-coverageshowmap_campaign_manifest.jsonrecords included/skipped artifacts so missing or invalid showmap files are visible