Release v0.5.0 with bug fixes, performance improvements, and docs update - #10
Merged
Conversation
From the React + Vite + Tailwind evaluation in .ignore/2.react-todo: five
models, five sessions, and src/App.jsx ended every one of them still holding
Vite's scaffolded counter demo. Components got written; nothing was ever wired
to anything. Almost none of it was the models.
Three shipped bugs, each of which silently degraded a whole feature:
judgeItem compared changeSet.size() across a TODO item. A ChangeSet is keyed by
path, so an item that edits a file an earlier item created leaves the map the
same size — the item was judged to have changed nothing, its done was
challenged, and a step that wrote a real file was reported as "it asked for a
file and none was written". Scaffold-then-assemble is the shape of every plan
worth making, so the item doing the work the user cared about was the one most
likely to be scored a failure. ChangeSet now carries a monotonic revision.
The completion check only challenges a done that changed nothing when the text
"requires a change", decided by a verb list written against messages people
type — and applied to text the planner writes. qwen3.5:4b planned "Assemble
App.jsx layout..." and "Configure exact folder structure...", neither verb was
listed, and both items passed unchallenged as done (no files changed).
requiresChange now takes { planned: true } with a separate vocabulary, kept
separate because "explain how the router handles a request" must still finish
having written nothing.
Challenged on a done, both qwen3.5:4b and ornith:9b replied by asking the user
what to work on, with the request still in the first message of the same
conversation. The objection now restates the ask verbatim.
Added — a read carries what the file imports. Reading App.jsx told a model a
hook was imported and nothing about what it returned, costing one turn per
import at CPU speed; qwen3.5:4b spent all 44 of its steps reading and listing
and wrote nothing. core/importGraph resolves local specifiers and read_file
includes them, capped, behind the same permission gate.
Added — step sessions, experimental and off by default, behind a header toggle
and hirayacoder.experimental.stepSessions. Each item is run as a briefed step
(agent/stepBrief) showing what the earlier steps actually wrote rather than only
that they finished; each is checked against its own text before it may close
(agent/stepGuard), since gemma2:latest edited vite.config.js and README.md while
working a list about useTodos, TodoInput and App.jsx; and a step that fails gets
one retry with the diagnosis stated, then stops the run with a workaround rather
than cascading through steps that depend on it.
Added — MemoryStore.readRelevant selects notes by subject before filling the
window by recency, path-aware so an item saying useTodos finds a note saying
src/hooks/useTodos.js. The step that had to assemble App.jsx ran sixth, by which
point the notes it needed were the first to fall out of the window.
Added — tools/bench-steps.js, the live benchmark for this failure. Neither
existing harness reproduces a project that already exists plus a multi-item
request whose last item must import what the earlier ones wrote. It resolves the
resulting imports rather than trusting that App.jsx names them.
Verified live on qwen3.5:4b, the model that failed before: 4 of 4 items,
App.jsx rewritten and importing all three files the run built.
Correcting the previous commit's closing claim. It said the live qwen3.5:4b run
had App.jsx "importing all three files the run built". That was measured with a
grader that only checked App.jsx *named* those imports on an import line, which
is too weak by exactly the margin that matters. Re-run with the specifiers
actually resolved, the same model wrote, from inside src/App.jsx:
import { useTodos } from '../hooks/useTodos.js';
import { TodoInput } from '../components/TodoInput.jsx';
Both climb one level too many, and TodoList was not imported at all. The right
files, the wrong route. The app does not build.
What the previous commit did fix stands: App.jsx is now rewritten at all, with
the counter demo gone, where five models across five sessions never touched it.
This is a narrower failure that was hiding behind that one.
Every guard passed it — the file is large, its brackets balance, it exports, no
body is a placeholder, the change set grew, and the file the step named is the
file that changed. Nothing in the project could see it.
importGraph.brokenImports resolves a written file's relative specifiers; bare
packages are excluded, since whether react is installed is a question about
node_modules rather than about what the model wrote. write_file appends the
failure to its own observation with the corrected path where exactly one file in
the workspace carries that name, and records it in detail.brokenImports.
Appended rather than refused, deliberately. The content is otherwise fine and
discarding a whole file over a path is where truncation and placeholder bodies
come from on a small model. stepGuard then reads the recorded result — no second
trip to disk, and no disagreement with what the model was told at the time — and
fails the step so the retry fires, with an instruction to correct the paths
rather than start over. Only the newest write per path counts, so a step that
wrote a broken file and then fixed it is not failed for the draft.
bench-steps.js now resolves the imports it grades and prints them verbatim, so
"attempted the wiring and got the path wrong" no longer reads as success.
createClient takes timeoutMs; requestTimeoutMs is the *setting* name and is silently ignored, so the harness ran at the 300s default rather than the 600s it asked for. On qwen3.5:4b that timed out mid-generation while rewriting App.jsx with four imports, and the step survived only because the retry caught it. Raised to 900s so the benchmark measures the model rather than the clock.
The React/Vite evaluation and all four bench-steps runs were produced on Machine A - the CPU-only Lenovo laptop that doc/MODELS.md calls the design constraint. Nothing said so, which makes the timings uncomparable and the conclusions weaker than they are: every model failing the same way matters more on the machine the project is shaped around than it would on a fast one. bench-steps.js now requires --machine, as bench-build.js already does. This task runs 20+ minutes on A and a fraction of that on C, so a result filed without its machine cannot be compared with anything. Recorded alongside it: the shipped hirayacoder.ollama.requestTimeoutMs default of 300000 is not enough on Machine A to generate one App.jsx with four imports. It timed out mid-run on qwen3.5:4b. That is an ordinary file on the machine this project exists for, so it is documented in doc/MODELS.md with a recommendation to raise it for laptop work. The default itself is left alone - it is also what makes a genuinely hung request noticeable on a fast machine, and B and C never approach it - so this is a documentation change, not a behaviour change. Run 3 is worth noting for the opposite reason: it completed only because that timeout produced a step that wrote nothing, the guard failed it, and the retry ran it again and succeeded. Under 0.4.0 the item would have ended empty. A slow machine and a step retry interact well.
Machine A cannot run the control. Its comparison is 0.4.0 code against 0.5.0 code, and those differ by three bug fixes as well as by step sessions - so it supports '0.5.0 wires the app where 0.4.0 never did' and not 'step sessions are what did it'. At 20+ minutes a run the paired steps/nosteps control was unaffordable; on Machine B it should be minutes. The handoff names the two things Machine A left open. The control itself, and whether the broken-import guard ever fires live - it has twelve unit tests built from Machine A's verbatim output, but no live run has produced a bad path since it was added, so it is unit-tested and not yet observed working. It also asks Machine B to settle whether the 300000 request-timeout default is only a laptop problem, and says plainly that 'both succeed' would be a more useful answer than a confirmation, so the run is not read as looking for one.
… Linux Found while writing the Machine C handoff, then confirmed on Machine A rather than handed off. importGraph resolved a written file's imports with fs.stat, which on Windows and macOS resolves ./hooks/usetodos.js to useTodos.js and reports success. So a model that gets the case wrong produces a file that builds locally and fails on Linux CI or a Linux deploy, and the guard added one commit ago would have called it fine - wrong in the one direction that ships a broken build. existsExactly reads each parent directory and compares every path segment byte-for-byte, because readdir returns the real spelling however the lookup was cased. Directories count as well as filenames: ./Hooks/useTodos.js is as broken on Linux as ./hooks/usetodos.js. findByStem still matches case-insensitively, which is what lets the suggestion carry the correct spelling back. Also records Machine A run 4: App.jsx rewritten, counter gone, all three imports resolving, 781s. Three of four runs now produce a wired app - but that is a rate from four samples, and the one failure predates the import guard, so it cannot be read as evidence the guard fixed anything. The guard still has not been observed firing live. Adds setup/FOLLOWUP-PROMPT-MACHINE-C.md. Machine C's job is the repeat count neither A nor B can afford: a success rate, and enough runs that the guard has a real chance to fire.
Machines B and C are benchmarking concurrently on this branch. bench-steps only printed to stdout, so both would have hand-edited the same region of doc/MODELS.md - the exact conflict bench-build's one-file-per-run rule exists to prevent. Same convention now: benchmarks/results/<machine>/, never appended to a shared file. The record carries the grade from the filesystem, the App.jsx import lines verbatim, and the model's own summary kept and clearly labelled as graded on nothing. tools/bench-steps-summary.js collates them into a rate. This matters more here than on the other benchmarks: one run of the wiring task tells you very little - Machine A ran the same model against the same fixture four times and got three different outcomes - and Machine C is being asked for five to ten repeats, which is a lot of terminal scrollback to collate by hand. It names each failing run and why rather than only reporting a percentage. A bimodal result is the interesting kind and averaging it away is the mistake both handoffs warn against. Verified end to end on qwen3.5:0.8b, which fails the task in under a minute and is therefore a cheap way to exercise the writer.
The README opened with "fully offline VS Code extension that pairs your editor with a local Ollama LLM", then went straight into tier tables, VRAM ceilings and a nineteen-run benchmark matrix. All of it true, none of it readable by the audience most likely to want a free private coding assistant: someone learning to code, or not coding at all yet. Restructured so the first half answers the questions a beginner actually has, in the order they have them. What is this. How is it different from ChatGPT. Will it run on my computer - answered by RAM, with instructions for checking how much you have. How do I install it, in four steps with what each one is for. What do I type first, with examples that work. Then a troubleshooting section written around what small models really do, since those failures look like the user's fault and are not. A refused write is the guards working and says so. "It said done but nothing changed" is a named habit with a fix. "It's taking forever" is a RAM problem with a smaller model as the answer. The honesty is deliberate and load-bearing: there is a comparison table that says outright that quality is "good, not great" against the paid tools, and a paragraph saying a request for a whole social media app will disappoint. Someone who expects Copilot and gets a small local model will otherwise conclude they did something wrong. Nothing was dropped. The constraints, benchmark tables, machine specs and recommendations moved under "For developers" or into doc/MODELS.md, which already held the full detail. Every factual claim was checked against the source rather than carried over: the keybinding, the VS Code floor, zero production dependencies, the Review diff button, deletes confirming even under Auto Edit, and the allow-listed binaries.
Machine A could afford four runs of this task and got three different outcomes, so it asked C for the thing four samples cannot give: a rate. This machine runs it in 30-65s, so 25 runs cost under half an hour. Ten pairs of qwen3.5:4b plus five gemma4:e4b, one at a time with nothing else running, every file written by the harness and committed unedited. All 25 reported 100% GPU, gemma4:e4b at 9.6 GB included. The two qwen3.5:4b failures are kept for the reason the handoff gives for keeping them: one wired every import correctly and left the counter demo in place while reporting 4 of 4 done, and one wrote every import a level too high and triggered the first live firing of the broken-import guard.
…n firing Answers the five questions in the Machine C handoff from the 25 runs in the previous commit, and adds a subsection alongside A and B rather than touching either. The headline is a negative result: 80% with step sessions against 70% without, at n=10 each, is indistinguishable. What the comparison does separate is 0.5.0 from 0.4.0 - where Machine A saw App.jsx untouched in five sessions out of five, the same nosteps path wires it 7 times in 10. The three bug fixes did the work. Also records that bench-steps-summary.js overstates the nosteps rate as 100%. Its predicate is wired.length > 0, so one resolving import out of three passes, against its own comment saying two out of three is a broken app. Three nosteps runs wired only part of the app - two never imported the hook holding the state. Reported rather than patched: Machine B is collating with the same tool right now, and moving a grading bar after seeing results is what section 5 forbids. The broken-import guard fired in a live run for the first time, reproducing Machine A's run 2. The retry did not rescue it - the model re-read all four files and wrote the same wrong paths - but the run ended partial and said "3 of 4 completed" instead of the confident 4 of 4 it gives elsewhere. Confirms the macOS half that had only ever run on Windows: the case tests pass on APFS, and the workspace boundary refuses traversal spelled with differently-cased components. Full suite 932 passing, 0 failing, so Machine A's four failures do not reproduce off OneDrive.
package.json is at 0.5.0 and package-lock.json still said 0.4.0, so npm install rewrote it. Committed on its own rather than folded into the benchmark commits.
Add Machine C results and update documentation for benchmarks
Machine C caught this by hand-counting 25 runs. The predicate shipped as (g.wired || []).length > 0 while its own comment claimed 'two out of three is a broken app that reports itself as finished'. The code won. Three nosteps runs had wired only part of the app - two imported TodoInput and TodoList and never useTodos, so the hook holding all the state was written and unused; one imported only useTodos, so neither component rendered. None had a broken path, only missing ones, so broken was empty and the check passed. A real 7/10 was reported as 100%. It failed in the worst possible direction. The inflation landed on the control arm, which is the one that makes the feature under test look bad by comparison, and the reading it produced was 'step sessions made things worse' from 25 runs that say no such thing. Corrected: 8/10 with step sessions against 7/10 without, which is indistinguishable at n=10. The bar is now what the comment always said, and it travels in each record as graded.expected so a future collator cannot quietly disagree with the task it is grading. Partially-wired runs get their own column instead of hiding among the passes - 'imported two of three' and 'imported none' are different problems. benchStepsSummary.test.js pins every shape those 25 runs produced, including the two that used to slip through, and asserts the collator reproduces Machine C's hand count. The module now guards its main behind require.main, like bench-build.js, so requiring it to test the grading half does not run a sweep. Re-grading after the fact is safe because the per-run JSON is the source of truth: nothing needs re-running, and Machine B - benchmarking right now - only has to pull and regenerate its table.
…oad data Two sweeps of four paired models, 16 runs, one at a time with nothing else running. The table is the second sweep on the tree of c38c424; the first ran on 7c8ab04, before the case-sensitivity fix, and is kept as a repeat rather than merged into the rate. The control Machine A could not afford says nosteps does not fail where steps succeeds. It never failed at all: all 12 Tier A runs rewrote App.jsx and removed the counter demo either way, and Machine A's motivating failure - components written, App.jsx untouched in five sessions out of five - did not reproduce once. Same conclusion Machine C reached from a distribution, reached independently on different hardware. The three bug fixes did the work. The pairs cannot separate the two arms and the second sweep proves rather than suggests it: two cells changed answer between sweeps with nothing altered but the run. Counting all 12 Tier A runs, nosteps is fully wired 5 in 6 and steps 3 in 6 - pointing opposite to C's 8/10 against 7/10 and equally insignificant (p ~ 0.55). Two machines disagreeing in direction at n~6-10 is what no effect looks like. What the pairs do separate is cost. nosteps was faster in all eight pairs, mean 255.0s against 297.6s: step sessions cost ~17% wall clock, and eight out of eight in one direction is a real signal where the correctness difference is not. The broken-import guard did not fire in 16 runs, and the reason is informative. Machine B produced zero broken paths - every miss was a missing import, models writing useTodos.js and then importing useState instead, leaving the hook holding all the state written and unused. The guard needs a path pointing at nothing; C saw exactly that and saw it fire. The failure has two modes and only one is guardable. The truncation and export-preservation guards did fire here. The retry never fired on B at all, 0 in 16. Its value rests on C's evidence. gemma4:e4b is the most interesting row: fully wired 3 of 4, indistinguishable from qwen3.5:4b, at 85% CPU / 15% GPU with 9.5 GB resident against a 4 GB card - and only 4% slower for a model 2.8x the size. A long generation is bottlenecked on the CPU-resident layers either way, so a badly-split model is not unusable on this task. No request timed out in any run; whole-session times were 211-348s and a session is many requests. Three machines now agree the 300s default is Machine A's problem alone. Suite is 947 passing, 1 failing. The 4 transcriptStore/scriptRunner timeouts predicted from Machine A do not reproduce, which with C's identical result confirms them as A's OneDrive-synced working directory. The one real failure arrived with the README rewrite in c38c424: the rewritten Requirements section says "(node, npm, python, git, and similar)", dropping gradle, make, java and javac from DEFAULT_ALLOWED_BINARIES, and the drift test catches it. Left unfixed - section 5 forbids code changes inside a benchmark run.
Closes the release with what three machines actually found, including the part that does not flatter it. Step sessions do not improve correctness. 33 paired runs of the wiring benchmark: Machine C 8/10 with against 7/10 without (p = 1.0), Machine B 3/6 against 5/6 (p = 0.55). Two machines disagreeing in direction, neither significantly, is what no effect looks like. The cost is the part that is significant - B's nosteps arm was faster in all eight pairs, about 17% less wall clock (sign test p = 0.008), exactly what an extra planning call plus one loop per item should cost. So the three bug fixes are what fixed the original failure, not the feature built on top of them. v0.4.0 left App.jsx untouched in five sessions of five; the control arm of these runs wires it 7 of 10 on C and 5 of 6 on B, and Machine A's motivating failure never reproduced on B. The feature ships off by default and labelled experimental, because it buys honesty rather than capability: a step checked against its own text, and a run that stops and explains itself instead of cascading. Machine C watched exactly that - the import guard fired on a real wrong path, the retry failed, and the run ended '3 of 4 completed' with the reason attached, where the same model unguarded reported '4 of 4' for an app that still had the counter demo in it. FEATURES.md now says outright to leave it off unless you want the stricter reporting. Also fixes a README regression the rewrite introduced and Machine B caught: runScriptTool asserts every DEFAULT_ALLOWED_BINARIES entry appears in the README, and the beginner rewrite replaced the toolchain table with 'node, npm, python, git, and similar', dropping java, javac, mvn, gradle and make. The full list is back, in the troubleshooting section where a beginner looks for it. 943 unit tests passing. The one failure is transcriptStore timing out on this machine's OneDrive-synced directory; B and C both confirmed it does not reproduce on their hardware.
jaymar921
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several important updates and experimental features to the agent's session handling and evaluation logic, with a focus on improving the accuracy of step completion detection, adding support for experimental "step sessions," and enhancing memory and import handling. The most significant changes include fixing a critical bug in how file changes are tracked, supporting more granular and robust step execution, and updating the changelog with detailed explanations of these improvements.
Core logic and bug fixes:
ChangeSetclass now tracks a monotonicrevisioncounter instead of relying on the number of unique files, ensuring that edits to files created in earlier steps are properly recognized as changes. This fixes a major issue where important edits could be missed and incorrectly reported as failures. (app/agent/agentSession.js) [1] [2]judgeItemfunction now uses thisrevisioncounter to determine if a step made progress, rather than comparing the number of files before and after. (app/agent/agentSession.js)Step sessions (experimental feature):
this.stepSessions, which:app/agent/agentSession.js) [1] [2]Memory and context improvements:
app/agent/agentSession.js)Import and file handling:
CHANGELOG.md)CHANGELOG.md)Documentation and benchmarking:
CHANGELOG.mdhas been updated with thorough explanations of all the above changes, including benchmarks, measured outcomes, and rationale for experimental features. (CHANGELOG.md)These changes collectively improve the reliability, transparency, and debuggability of the agent's step execution and evaluation process.