Improve script execution and error handling in builds v0.6.0 - #12
Merged
Conversation
Commands started at the workspace root and nowhere else, so a project scaffolded into a subfolder could never be built. The only phrasing models had for it — `cd app && npm run build` — is refused as chaining, and the refusal named no alternative, so they resent it until the repeat guard ended the step. Across the v0.5.3 model round this was the single largest cause of "failed to run scripts", and it had nothing to do with model size. run_script now takes an optional cwd, resolved through the same pathGuard confinement reads and writes use, and taken from the approved decision rather than the request so it cannot be swapped in between the click and the spawn. It is declared as an optional field in the action schema, without which constrained decoding leaves Tier B models unable to emit it at all — the same trap `recursive` was in. It joins the repeat-guard key, the step trace, and the memory notes, so two installs in two folders are two actions, and a later step builds where the earlier one did. Also close stdin and set CI=1 on every spawn: `npm create vite@latest` asks whether to proceed, and a process blocked on a question nobody will ever answer was indistinguishable from a hung build until the timeout killed it.
A non-zero exit code is not a diagnosis. Handed `exit code 1` and 400 tokens of npm output, small models resent the identical command, announced the build had succeeded, or abandoned a task they were two steps from finishing — all three seen in the v0.5.3 round. Every failed run is now classified into one named reason — missing dependency, wrong path, syntax error, permission, environment, network, port in use — and each reason carries the one sentence the model can act on, placed last in the observation so it is the final thing read before deciding. Exactly two reasons are retried, once: a network blip and a file lock, where the command was right and the world was briefly wrong. Nothing else is. The damage in the testing round came from retries, not their absence, so a missing dependency is told to install rather than quietly run again. Servers no longer eat the whole budget. `npm run dev` succeeding looks identical to `npm run dev` hanging, and every model asked to "confirm the dev server starts" spent two minutes there and read the kill as a failure. Those commands now get a 20-second probe: still up and quiet means it started, and the model is told not to run it again. A server that printed EADDRINUSE or could not resolve a module still fails, and says which. Pre-flight answers the provable cases from the filesystem before anyone is asked to approve anything: no package.json in that folder, no such script (with the list of scripts that do exist), dependencies never installed. Each costs a stat instead of a click, a subprocess, and a page of output.
Closing a chat tab called session.cancel(). That is the right answer for a turn still queued and the wrong one for a turn in flight: on CPU inference a turn is minutes long, so the agent is regularly mid-build when someone closes the wrong tab, and everything it had done went in the bin at the moment it could least afford it. A running turn now detaches instead. The run continues headless — permission prompts are VS Code modals rather than webview panels, so it can still ask — and the transcript records the outcome, so reopening the session shows what happened whether or not anyone was watching. A notification offers Reopen or Stop, which is the explicit end the run does respect. A queued turn still gives up its lane, because it has done nothing worth saving and holding the lane starves every other session. Reopening a detached session repaints the conversation and then says the turn is still going, instead of showing an idle composer over a session that is still writing files and refusing the next message with no visible reason. The panel handlers move onto the tab, since a detached session opens a second panel when it is reopened and the registry wiring was attached only to the first — the tab would never have been forgotten and focus would have stopped following it. Closing the window still stops everything: the extension host is going away, and an orphaned npm install has nothing left to read it.
…mbers in front of you Three things that all come back to a small model losing the thread. The goal is now restated last, immediately before the instruction to act. contextBuilder already puts the task at the top of a block rebuilt every turn, which is correct and is not enough: by the time a 1B model has read a project overview, a file listing, session memory, a step trace and 400 tokens of npm output, the sentence saying what it is for is thousands of tokens behind it, and recency wins. The step count rides along, because "you are on step 6 of 8" is what turns "keep exploring" into "write the file now". Sixty tokens a turn, the cheapest thing in the prompt. A `done` is now challenged when the task names three or more files and some of them do not exist. Three paths is a structure being specified rather than a sentence that happens to mention a file, and the benchmark shape — eleven files asked for, four written, "done" — was invisible to every other check. Existence is tested against the workspace, not just against what the run wrote, so files that `npm create vite` produced count as produced. The ledger records parameter count alongside the timings it has always kept, and the adaptation report finally shows both: average and slowest turn, time per action, and what share went to waiting on the model, ordered smallest model first. `timings()` has existed since 0.5.0 and was called from nowhere, so "gemma4:e4b took about thirty minutes" was a thing you could only learn with a stopwatch. Failed commands now carry their diagnosed reason as the error code, so the failure counters say MISSING_DEPENDENCY rather than 37.
…behind a slow run Logging fills the three gaps that made the v0.5.3 round hard to review after the fact: both ends of every TODO item rather than only the closing one, the step transition with its budget and whether the experimental brief was in use, and one debug line per action with its outcome and duration — the record that settles "it said it edited the file and it did not". The translator now logs the note it stored, not only the ones it threw away; a stored note comes back as trusted context on every later turn, and it was the one decision going unrecorded. CHANGELOG and FEATURES cover the round: the workspace-root limit on commands, the scaffolder prompt that looked like a hang, failure reasons and the two retryable ones, the dev-server probe, pre-flight checks, tab close no longer killing a run, the goal restated at the decision point, the named-file completion check, and the performance comparison in Show Learned Adaptation. 0.5.3 is left as its own unreleased section, since it is the build the round was run against and its notes are the record of what that build got wrong.
The 0.6.0 build packaged at 22.69 MB against 0.37 MB for 0.5.3 — 3,749 files
of React tooling, none of which this extension has ever imported. It is the
same root cause as the rest of this release: the model round could not run a
command anywhere but the workspace root, so `npm install` for the TODO app
went into the extension's own tree. package.json had already been cleaned back
to `"dependencies": {}`; the packages themselves were still sitting in
node_modules, and vsce packaged what it found.
`npm prune` is the fix. Excluding node_modules outright is the guarantee: with
no runtime dependencies there is nothing under it that belongs in a shipped
build, and a stray install should cost a rebuild rather than a release. The
line carries the warning that adding a real runtime dependency means removing
it. Repackaged: 84 files, 394.9 KB.
The lock file picks up the version bump.
Caught on `gemma3:1b` in a live build run. It ran `node src/main.js` before it had written main.js, Node answered Error: Cannot find module 'C:\…\hiraya-build\src\main.js' and the new diagnosis told it to run npm install — for a file it had simply not created yet. The model went and read a different file instead, and the phase ended on the repeat guard. Node reports a missing file and a missing package with the same four words. The rule tried to separate them by excluding a leading dot or slash, which a Windows absolute path sails straight past on its drive letter. Both cases are now decided on the specifier: a separator anywhere means a path the code named, and only a bare word — or a bare @scope/name — is a package. Vite's "Failed to resolve import './Foo'" moves across for the same reason, and MODULE_NOT_FOUND joins the path rule as a backstop. The wrong answer was worse than no answer here: it sent a model that was one write_file from working towards an install it did not need.
From a live 0.6.0 run of `gemma4:e4b` on the TODO-app spec. It installed, scaffolded, and wrote every component correctly — the cwd fix worked, npm ran where it was supposed to — then `npm run dev` died in 1.3 seconds: VITE v5.4.21 ready in 2912 ms [Failed to load PostCSS config: [ReferenceError] module is not defined in ES module scope … at postcss.config.js:1:1] `npm create vite` writes "type": "module"; the model wrote postcss.config.js with `module.exports`. A two-line fix. Instead the run reported the app finished, because nothing told it otherwise. Three gaps, all of them ours: **No rule matched.** A ReferenceError is not a SyntaxError, so `diagnose` returned null and the model got forty lines of Vite stack trace with no sentence saying what to do. The ESM/CJS mismatch — the single most common way a Vite + Tailwind scaffold fails — now has its own reason, naming both repairs (`export default`, or the same content as `.cjs`). **A server that exits still read as unexplained.** Vite prints "ready in 2912 ms" and *then* dies, so the most recent line in the output is the one saying it worked, which is exactly what a model believes. Any long-running command that exits at all is now a failure that says so, whatever it printed on the way out. **The loop offered the wrong next move.** For a failure in a file the model itself wrote, "use a different action, or finish with done" talks straight over an observation that has already named the file and the fix. Diagnoses carry `fixFirst` now, and those get told to open the file, correct it, and run the same command again — not to start something else and not to finish. And the structural half, which does not depend on the model reading anything: a `done` is challenged once while the last command it ran is still failing. A project whose build does not run is not finished. A run that broke the build and then fixed it passes untouched — that is the behaviour being asked for.
…ing a broken server Three findings from a live qwen3.5:4b run of the TODO-app spec, all of them ours. **npm installed outside the workspace.** The workspace was .ignore/0.6.0-todo-app-qwen3.5-4b, the project was in todo-glass-app/ inside it, and every npm command carried the right cwd except one: 13:52:02 "command":"npm install --save lucide-react" ← no cwd There is no package.json at that workspace root, and npm searches *upwards* until it finds one. It left the workspace, left .ignore/, and installed the dependency into this extension's own package.json. Exit code 0, reported as a success, three directories outside anything the user had opened — and it is where the tailwind/lucide pollution in this repo came from, twice. Path confinement binds the agent's tools; a subprocess resolves paths however it likes, and nothing here sees npm's search happen. Pre-flight now performs that search itself, bounded at the workspace root, and refuses any package-manager command whose manifest would resolve outside — naming what to set cwd to instead. A manifest in a parent folder that is still inside the workspace, as in a monorepo, is allowed: that one is the user's own. **A dev server serving nothing but 500s counted as started.** The probe asked "did it fall over?" against a list of specific failures — a port collision, a missing module, a config that would not load. Vite starts fine with a broken PostCSS config, announces "ready in 906 ms", and then fails every request. No pattern matched, so the probe reported a working server. The test is now the general one: did it say "error". A clean start does not, and "0 errors" is excluded so a build announcing its own success is not misread. **An error nobody wrote a rule for.** A rule list only covers what someone has already seen. Tailwind 4 moving its PostCSS plugin to @tailwindcss/postcss gets its own rule, since it fires on essentially every Vite + Tailwind scaffold a model writes. Behind it sits the general answer: nearly every build error names the file it choked on, so when nothing matches, the first project file in the output is extracted — skipping the frames that run through node_modules — and the model is told to open that file, change what the error describes, and run the command again. No classification needed. The old pre-flight test asserted that `npm install` with no manifest in scope was fine. That assertion encoded the bug and is now the escape suite.
Five models against the build-a-TODO-app benchmark, JavaScript only. The three at or below 1.2B score 0/3 — qwen3.5:0.8b and gemma3:1b never write both files and end on the repeat guard, llama3.2:1b creates a *folder* called src/todo-app.js and tries to run it. qwen3.5:2b passes all three phases in 35 minutes, which is the first full pass this benchmark has recorded from a model this small. Recorded as-is. The 0.5.3 baseline these should be read against has not been run yet, so none of this is an improvement claim — and the two script fixes that landed after them (1be0524, 3d4d370) are not in these numbers either.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and new features to the agent session logic, focusing on better tracking of command execution, improved logging for debugging and evaluation, and enhanced checks for file existence. These changes support more robust session management and provide clearer insights into agent actions and outcomes.
Agent session and command tracking enhancements:
commandsarray to the session summary, recording every command executed during a run (includingcwd), so that a "done" cannot be accepted if the build failed during the session. This helps ensure that failures are not silently ignored in subsequent steps. [1] [2] [3] [4]recordCommandmethod and related data structures now support an optionalcwdproperty, improving traceability of where commands were executed. [1] [2]File existence checks:
_existsInWorkspacemethod to safely check if a file exists at a workspace-relative path, using the same path confinement as other file operations. This is exposed to completion checks to validate that files named in tasks actually exist. [1] [2]Logging and diagnostics improvements:
Session metadata enhancements:
params) for more granular reporting and analysis. [1] [2]Dependency and ignore file update:
.vscodeignoreto always excludenode_modules/, clarifying that the extension has no runtime dependencies and preventing accidental packaging of development dependencies.These changes collectively improve the reliability, observability, and maintainability of agent sessions and command execution tracking.