fix: scorer.sh --update-queue overflows argv on large function counts - #156
Merged
Conversation
added 3 commits
July 30, 2026 08:25
jq --argjson entries "$(...)" passes the entire scored-entries JSON as a single argv string. At swkotor.exe's queue scale (12,845 functions with per-entry score/reason/metrics), this payload is multi-megabyte and exceeds the OS argv-size limit, hard-failing with "Argument list too long" -- this silently aborted the autonomous vacuum/repair campaign's queue-scoring step before it could run, discovered when a full-queue --autonomous run exited after only 15/15 stage-resume receipts with no vacuum activity at all. The same script already uses the safe pattern (--slurpfile reading from a temp file, avoiding the argv limit entirely) three lines above for the same kind of large-array plumbing -- this just applies it to the second call site that didn't. Verified: reproduced the exact "Argument list too long" failure with the old --argjson approach against a realistic ~3.6MB synthetic entries payload (12,845 entries), confirmed the --slurpfile fix runs cleanly against the same payload and produces correct merged/scored queue output.
Mirrors queue_tmp's existing pattern -- both temp files are only used inside the --update-queue block, so both should be lazily created there rather than unconditionally near the top for every scorer.sh invocation. Found by code review (ce-code-review, 6 personas); no correctness impact, just avoids a needless mktemp on every non-update-queue run.
Pre-existing on master, unrelated to this branch's scorer.sh fix -- inherited from the base commit. Fixed here since it was blocking this PR's ruff CI check. Same class of pre-existing lint issue fixed independently on several other branches this session.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Found while trying to re-run the autonomous vacuum/repair campaign for swkotor.exe (12,845 functions) after shipping the packaged-source stdcall/callee-calling-convention fixes in PR #155 -- the campaign exited after all 15 pipeline stages resumed cleanly but before any vacuum activity, with:
Root cause:
jq --argjson entries "$(printf '%s\n' "$report" | jq '.entries')" ...passes the entire scored-entries JSON (score/reason/metrics for every function) as a single argv string. At this queue's scale that payload is multi-megabyte, exceeding the OS argv-size limit.Fix: write
.entriesto a temp file and use--slurpfileinstead of--argjson-- the same script already uses this exact pattern (--slurpfile entries "$entries_tmp") three lines above for the same kind of large-array plumbing; this just applies it consistently to the second call site.Test plan
--argjsonapproach against a realistic ~3.6MB synthetic entries payload (12,845 entries)--slurpfilefix runs cleanly against the same payload and produces correct merged/scored queue outputbash -n scripts/scorer.shsyntax check passesqueue_tmppattern)Follow-up (not in this PR)
scripts/vacuum.sh:190callsscorer.sh --update-queuewith no logging around it, which is why this bug's failure was silent. Worth wrapping in the samelog_progress/write_sessionpattern used elsewhere invacuum_start().--update-queueat scale (repo has no shell test harness for this script).🤖 Generated with Claude Code
https://claude.ai/code/session_01Ros3797gzvmswnJudQ1Znk