Fix evaluation session defects and enhance agent functionality - #8
Merged
Conversation
Six conversations across two workspaces, building the same TODO app in Java, Python, then HTML, on deepseek-coder-v2 and ornith:9b. Three of these four bugs had shipped unnoticed since the features they belong to were written, and each degraded a whole feature silently rather than failing loudly. - chatTab's permissions button called modes.toggle(), a method PermissionModes has never had. Every click threw a TypeError into an unhandled rejection, so both permissions were unreachable from the chat tab. Delegated to the one menu that can actually apply a change, which is also the only path carrying the confirmation that enabling auto-approve-scripts requires. - Switching model took two clicks: setModel only writes the setting, and the listener that adopts it is fire-and-forget, so the repaint read the previous activeModel and drew the dropdown back. selectModel now adopts the setting and awaits its own refresh, and refreshes are serialized so the two a single model change produces cannot settle out of order. - Plan mode's checklist depended on the closing `done` summary happening to be a numbered list. A run that ends on the repeat guard has no `done` at all, so the plan was empty and "Run this plan" never appeared. The plan is now asked for directly when the summary yields nothing, and falls back to prose rather than inventing steps. - Added create_folder and delete_folder. A folder could not be removed by any route the agent had — delete_file refuses directories and the rmdir redirect pointed at it — and a model with no way out narrates one: asked to remove an empty src/main/java, it reported the folder deleted when it was still there. delete_folder is empty-by-default, always confirms in every permission mode, and refuses past 100 entries. 699 unit tests passing (+29).
…learns The second half of 0.4.0, and the part the evaluation session was really about. Agent mode constrains Tier B decoding to a grammar whose every branch is a tool call, so a greeting could not produce a greeting — the model's only legal move was to pick a tool. Nine conversational messages in one session came back as nine variations of "I stopped because I kept repeating the same step". core/intentRouter now classifies each message first, and a conversational one is answered directly: one call, no loop, no tools. task is the default and chat needs positive evidence, since being wrong toward task costs a loop while being wrong toward chat silently drops a request. Any imperative overrides it, so "hi, can you fix app.js" is work. chatTab.history was display state that the model never saw, and the transcript on disk was written and never read — which is why "can you remember our first conversation?" was answered by searching the workspace. Earlier turns now reach the prompt as their own budgeted section, on working turns as well as conversational ones, ranked above session notes because the notes are a compression of the same material. core/factStore holds what memory structurally could not: typed, workspace-scoped facts rather than a log of actions. Session 1 spent its whole budget finding out javac could not run; session 2 spent its budget finding out again and then proposed apt-get on macOS. Detection is pattern-matching over what a program printed and involves no model call, because a wrong fact persists and is stated to every future turn as settled. All three platforms' phrasings of a missing toolchain are recognised, including Apple's stub javac that no PATH check could have caught. 746 unit tests passing (+47).
Two ways a run reported success without producing any. It never wrote anything: asked five separate times to convert a Python app into todoapp.html, the agent replied "2 of 2 item(s) completed … done (no files changed)" every time, which is accurate and reads as success. Or it wrote a placeholder: the file that eventually appeared had both handlers as a comment and a console.log, so a change set grew and nothing downstream had reason to doubt it. agent/completionCheck runs when either loop is told the work is finished and can send the model back once. Once, never twice — a model that cannot produce the work will not be argued into it, and refusing indefinitely burns the budget to reach a worse report than the honest one. Narrow at both ends: a request that only asked to look at something is never challenged, a // TODO in working code is not a placeholder, and Plan mode is exempt since changing nothing is its purpose. Also fixes a quadratic regex found while checking the detect-unsafe-regex warnings instead of waving them through: two \s* either side of an optional group, 308 ms on 20k spaces against 0.17 ms fixed, on a pattern that scans every file the agent writes. The other five warnings were measured and are false positives. 766 unit tests passing (+20).
Conversational routing in Agent mode, the three memory layers and what each one answers, the completion check, and the folder tools with the reasoning behind delete_folder being the one mutation with no auto mode at all. The architecture table gains intentRouter, completionCheck, and factStore, and the data-on-disk section explains why session memory, the transcript, and facts are three files rather than one: what the agent did, what was said, and what is true — the last of which is workspace-scoped precisely so a second session does not pay again for what the first one discovered.
…orpus The .vsix was shipping .ignore/, which is where local evaluation transcripts live — 106 KB of real session data including the absolute paths of the machine it was recorded on. .gitignore covers that folder, and a .vscodeignore file does not inherit those rules, so it was invisible to git status while being packaged into a distributable file. Anything untracked is more likely to be private, not less. .claude/ and benchmarks/ went the same way: neither is read by anything under app/. This is the third instance of the same class of bug — the tools/** entry carries a note about shipping "because only source folders were excluded" — so each new entry says why it is there rather than just listing a path. 98 files → 73, 312 KB → 284 KB. Verified against the built archive: no .ignore, .claude, benchmarks, test, scripts, doc, or security paths, and setup/prompts plus both icons still present, which are the runtime files an over-broad exclude would quietly drop.
Testing on ornith:9b against the same Java → Python → HTML prompt. "okay proceed" was answered as small talk. The pleasantry rule matched a social word at the start of a message under a six-word cap, and that phrase satisfied both — so the model got no tools, replied with the complete HTML in a code fence, and wrote "Saved to todoapp.html." Nothing was saved and the user asked three more times. This is exactly the failure the module's own header warns about. A social word at the front says nothing about the rest, so the test is now that the whole message is social. Assent words are excluded and the omission is a named constant: "ok", "sure", "proceed" mean carry on, and dropping that request is far worse than spending a loop on it. An unanswered completion challenge was reported as success. The check fired, the model said done again, and the summary the user got was the word "Finished." The second done is still accepted — arguing with a model that cannot do the work is worse — but the summary now says nothing was created, edited, or deleted, and a TODO item in that state drops to failed. A program that prints "coming soon" now counts as unfinished. TodoManager.java was written correctly with real add/remove/modify; TodoApp.java never constructed one and printed "Add feature coming soon." for every menu option. Both compiled, the change set grew, and nothing in the system could see it — there is no comment, no empty body, and writeFile's guards are about damage rather than hollowness. Matched at file level since the Java version buried it inside a switch inside main. A bare TODO in a string is not a match: the program this was found in is a todo app. The permissions menu now names the prompts each toggle covers. Auto Approve Running Scripts was on and the Create/Apply dialogs kept appearing, which reads as broken; those are Auto Edit's, and the two are independent. 773 unit tests passing (+7).
Both local, in the ledger that already exists for this. Durations are numbers and states are enums, so outcomes.jsonl keeps the property that makes it safe to keep: no paths, no commands, no content. Timing lives in ollamaClient.request, the single funnel every call passes through, so no call site has to remember to measure. Each turn logs wall-clock and how much of it was spent waiting on the model — "94.2s (96% waiting on the model)" — because a four-minute turn is a different bug depending on which. The session figure is a subtraction across the client's running total rather than a sum of instrumented loops, which is what catches the planning and TODO-splitting passes that happen outside any loop and are often where the time went. Steps are timed too, including confirmation waits: a session slow because a dialog sat unanswered is not a slow model. Health is three states rather than a boolean, because the two failures need opposite responses. down means nothing is listening and is called on the first failure, since a refused connection will not become truer on a second. unresponsive means it accepted the connection and then did not answer twice running — the wedged case, where restarting helps — and needs two because a large model loading into memory legitimately blows a deadline. A 4xx leaves it up: the request was wrong, not the server, and reporting that as an outage sends the user to restart something that works. Cancellations count for nothing; pressing Stop is not evidence. Transitions are recorded, not requests, so a healthy server costs nothing and a flapping one reads as a few timestamped flips. Notifications only on entering a state the user can act on. Show Status has the glance version. 792 unit tests passing (+19).
…istory From the v0.4.0 test across deepseek-coder-v2, ornith:9b, gemma4:e4b and qwen3.5:4b. Tier A ran a tool call with no path. Tier B validates required fields and refuses with a correction naming what is missing; Tier A trusted Ollama's structured format and passed an argument object without `path` straight to write_file, which resolved `undefined` and answered "The write to undefined was not applied." gemma4 sent five identical writes, then told the user the tool environment was broken and reported the file as written. It was right that something was broken and wrong about what. Both tiers now validate against the same REQUIRED_FIELDS table. The assent-word exclusion was over-corrected. Excluding "ok"/"sure"/"proceed" was right; making any message containing one a task was not — "okay thank you" ran a four-item TODO list that re-analysed five files. Assent is now neutral: alone it means go ahead, with a real pleasantry it means acknowledgement. Also fixed "how are you" (read two source files and reported on them), "hello gemma4" (a model name is in no vocabulary and never could be), and questions about where the work has got to, which contained "verify" and were claimed by the work-verb rule. A UI wired to an empty script now counts as unfinished: 52 lines of real markup, four styled buttons, and a script block holding one comment. No placeholder string, no function bodies to be placeholders, change set grew. Narrow by design — static pages, external scripts, inline handlers and single controls are all left alone. Also dropped `placeholder` from the literal word list; it is a standard HTML attribute and fired on ordinary form markup. core/fileHistory records a bounded diff per write, so what a file looked like before is recoverable. Show File History renders them as a diff document. The agent gets paths and counts only, under "files you have already changed in this session — do not redo this work", which is the half that changes behaviour: a model asked to modify a file it edited three turns ago has no idea it did, and undoes its own work. Diffs not snapshots, since snapshots duplicate git. 824 unit tests passing (+59).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request enhances the agent session logic to more robustly verify that tasks are truly completed, improves user feedback when nothing is produced, and upgrades plan mode reliability for small models. It also introduces new configuration options, records more detailed session and step metadata, and expands
.vscodeignoreto avoid leaking local or private files in extension packages.Agent completion verification and user feedback:
Plan mode improvements for small models:
Session and step metadata tracking:
Intent classification and conversational mode:
Packaging and documentation:
.vscodeignorefile is expanded to exclude local scratch space, agent tooling, and benchmark data from extension packages, preventing accidental leaks of private or development-only files. The README is updated to document these new behaviors, including conversational awareness and stricter "done" verification. [1] [2]