Skip to content

feat(harness): add task-scoped experience traces - #77

Closed
U1oSun wants to merge 2 commits into
QoderAI:mainfrom
U1oSun:feat/lc03-experience-trace-v1
Closed

feat(harness): add task-scoped experience traces#77
U1oSun wants to merge 2 commits into
QoderAI:mainfrom
U1oSun:feat/lc03-experience-trace-v1

Conversation

@U1oSun

@U1oSun U1oSun commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add better-harness harness experience-trace create|validate as the first executable LC-03 slice.
  • Project one explicit Qoder report.source.json into deterministic, versioned canonical JSONL.
  • Bind caller-owned task, workspace, and run keys without emitting the raw keys; require an exact Episode ref or explicit no-session evidence.
  • Expose honest partial/unavailable coverage, stable private errors, root CLI discovery, public exports, and package/runtime-bundle ownership.

Why

Traceability and Scope

  • Spec/ADR: docs/specs/2026-08-11-lc03-experience-trace-v1.md
  • Acceptance criteria: ET-AC-1 through ET-AC-10
  • Canonical owners changed: scripts/experience-trace/, root CLI registry/dispatch, architecture routing, npm/runtime pack verification, tests, fixtures, and changelog
  • Explicit non-goals: no full cross-host LC-03, Checkpoint persistence, LC-02 snapshot binding, LC-05 mining changes, runtime/OTLP emission, gate or delivery decisions

Change Type

  • Feature
  • Bug fix
  • Tests only
  • Documentation/community
  • Refactor with no intended behavior change
  • Dependency, packaging, or infrastructure

Test and Review Evidence

Check Result
node --test test/experience-trace-contract.test.mjs test/experience-trace.test.mjs test/experience-trace-cli.test.mjs test/better-harness-cli.test.mjs test/scripts-refactor-contract.test.mjs 82 passed, 0 failed
node --test test/harness-report-source.test.mjs test/task-loop-source.test.mjs test/session-episode-contract.test.mjs 46 passed, 0 failed
node scripts/doc-link-graph/cli.mjs skills/better-harness + doc-link test graph regenerated; 6 passed, 0 failed
npm run pack:verify passed; npm 473 entries, runtime zip 495 entries
npm test 1,323 total; 1,322 passed, 0 failed, 1 skipped
git diff --check passed
Independent adversarial review 3 Terra Max reviewers: verdict=pass, p1_p2_clear=true

Manual or visual evidence: both versioned JSONL fixtures reproduce byte-for-byte with independently replayed identities and digests. No visual change. No CI status is claimed.

Risk and Recovery

  • Compatibility and cross-platform impact: argv-based Node CLI with no shell-specific command construction; deterministic UTF-8/LF contract tested on Windows and designed for Windows, macOS, and Linux.
  • Package, plugin, schema, or generated-file impact: adds Experience Trace v1 modules to both npm and Qoder runtime bundle; adds two normative JSONL fixtures and frozen CLI output updates.
  • Rollback or recovery path: revert commit 560be0e; the feature is read-only and writes no artifacts.
  • Residual risk or unverified boundary: caller-to-business-task truth remains caller-asserted. External API Maker execution and native Qoder runtime proof were not performed; the versioned fixture covers the unavailable-evidence boundary.

AI Involvement

Level: Assisted.

Human review and validation: automated evidence and two independent counterexample-review rounds are complete.

Checklist

  • I followed AGENTS.md, CONTRIBUTING.md, and the relevant canonical-owner guidance.
  • The change is focused and does not include unrelated local or generated state.
  • Tests and documentation match the behavior actually delivered.
  • Markdown links were checked when documentation moved or changed.
  • Cross-platform behavior was considered for Windows, macOS, and Linux.
  • Package/runtime verification was run when shipped files or dependencies changed.
  • User-facing or compatibility changes are recorded in CHANGELOG.md.
  • I have the right to contribute this work under the repository's MIT License.

Implement the first executable LC-03 slice with a bounded, privacy-safe Qoder source projection, canonical create and validate JSONL contracts, maintainer CLI discovery, golden fixtures, and package checks.

Issue QoderAI#70 is related consumption evidence and is not closed by this change. Bindings remain caller-asserted, and partial or unavailable traces remain advisory-only. Validation covered focused, cross-module, documentation, package, runtime-bundle, and full regression checks.

Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the first LC-03 task-scoped Experience Trace capability, producing deterministic, privacy-safe JSONL from Qoder report sources.

Changes:

  • Implements bounded trace projection, serialization, validation, and CLI routing.
  • Adds normative fixtures and extensive contract, privacy, bounds, and CLI tests.
  • Documents architecture, specification, packaging, and user-visible behavior.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.

Show a summary per file
File Description
scripts/experience-trace/contract.mjs Defines the canonical trace contract.
scripts/experience-trace/project-source.mjs Projects validated Qoder report sources.
scripts/experience-trace/cli.mjs Implements create and validate commands.
scripts/experience-trace/command-manifest.mjs Declares command metadata.
scripts/experience-trace/index.mjs Exposes the public module surface.
scripts/better-harness-cli/registry.mjs Registers the new subcommand.
scripts/better-harness-cli/cli.mjs Delegates strict help validation.
scripts/npm-package/verify-pack.mjs Verifies packaged trace modules.
test/experience-trace-contract.test.mjs Tests wire contract and bounds.
test/experience-trace.test.mjs Tests source projection behavior.
test/experience-trace-cli.test.mjs Tests CLI behavior and privacy.
test/better-harness-cli.test.mjs Tests root CLI integration.
test/scripts-refactor-contract.test.mjs Updates discovery output hashes.
test/fixtures/scripts-refactor-contract/root-help.txt Updates root help fixture.
test/fixtures/scripts-refactor-contract/harness-help.txt Updates harness help fixture.
docs/specs/fixtures/lc03-no-session-v1.jsonl Adds unavailable-evidence fixture.
docs/specs/fixtures/lc03-episode-v1.jsonl Adds Episode trace fixture.
docs/specs/2026-08-11-lc03-experience-trace-v1.md Specifies LC-03 V1 behavior.
docs/ARCHITECTURE.md Records capability ownership.
CHANGELOG.md Announces the new command.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Keep Experience Trace golden fixtures byte-identical on Windows by declaring JSONL files as LF-normalized text. This follows the cross-platform fixture contract in docs/specs/2026-08-11-lc03-experience-trace-v1.md for roadmap LC-03 and related issue QoderAI#70, and was validated with focused tests plus npm test.

Co-authored-by: Codex (GPT 5.6 Sol) <codex@openai.com>
@phodal

phodal commented Aug 12, 2026

Copy link
Copy Markdown
Member

Thanks for this, and apologies for a missing-context problem that was on my side, not yours.

Context you could not have had

When you branched from 81440ba, two commits were sitting unpushed on my local main. They are on origin/main now, as of 67d6d1e. The relevant one is 03267cbscripts/commit-session-link/, spec docs/specs/2026-08-11-commit-session-correlation.md.

Please take a look at it before deciding what to do with this PR, because it already implements the commit-link half of the LC-03 acceptance row.

What commit-session-link does: answers "which agent sessions produced this commit" from local evidence only. Its design inputs were the entire.io commit/session pages and the entireio/cli domain model, while explicitly rejecting that project's shadow-branch, refs backend, and checkpoint-first product model.

Module Responsibility
git-facts.mjs Bounded commit metadata plus git diff-tree --numstat -z; hash, subject, author/committer time, per-file +/-, typed session/checkpoint trailers parsed from the body
entire-checkpoints.mjs Read-only resolution of allowlisted checkpoint ids (12-hex or 26-char ULID) into checkpoint-to-session facts; no fetch, checkout, ref update, or hook write
correlate.mjs Pure scoring: session summaries × commit facts → ranked matches with per-match evidence
session-source.mjs Bounded hydration via createAnalyzer(platform); falls back to a resolved Entire transcript when native discovery cannot find an explicitly selected session
session-report-model.mjs SessionViewerReportV1, the single renderer-facing read model
render-html.mjs / render-session-html.mjs Commit view plus Session Viewer with activity rail and bounded tool-call trace

The part that overlaps this PR is the confidence ladder in correlate.mjs:

explicit  <- a Harness-Session: trailer names a discovered session id, or an
             Entire-Checkpoint: trailer resolves through read-only checkpoint
             metadata to that session id
high      <- time overlap plus at least one overlapping repo-relative file
medium    <- time overlap plus session cwd inside the repository
low       <- time overlap only
excluded  <- no time overlap and no explicit trailer

Grace window defaults to 45 minutes after lastSeen; committer time drives heuristic correlation while author time is retained for display.

That is the roadmap.md:154 clause "Commit links distinguish explicit evidence from bounded heuristic candidates", and roadmap.md:174 step 1 of the First Coherent Vertical Slice: "Link one Task Episode to one local Commit with explicit or clearly labelled heuristic evidence. Timestamp proximity alone is insufficient."

Where that leaves this PR

I do not think this slice should merge in its current shape, for three reasons that are architectural rather than stylistic.

1. Two canonical lists over one fact domain. CAPABILITY_ORDER with its observed/derived/unavailable × source-projected/caller-asserted/unavailable taxonomy, and CONFIDENCE_LEVELS with explicit/high/medium/low, are two independent authoritative classifications of the same thing: evidence binding Task Episodes to changes and outcomes. ADR-0002 "Keep facts with federated canonical owners" is explicit here:

When two surfaces need the same structured fact, they must consume or validate the same declaration. They must not establish a second canonical list in a test or documentation file. [...] The catalog rejects missing owners, multiple authoritative contributors for one field, and contradictory values; it has no precedence rule that lets it choose a winner. Conflict resolution happens in a dated spec owned by the conflicting capabilities before projection resumes.

So the required next step is a dated spec jointly owned by experience-trace and commit-session-link that settles the shared ref vocabulary and the provenance/confidence taxonomy. That is fail-closed routing, not a preference.

2. Evidence class taxonomy is routed elsewhere. ADR-0001's routing table assigns "Shared evidence envelope, class taxonomy, or redaction invariant" to scripts/evidence-contract/. This PR builds a new availability/provenance taxonomy and its own redaction boundary inside scripts/experience-trace/contract.mjs (922 lines). ADR-0002's target-ownership table records that fact class as having no shared current owner, state planned. This adds a third producer-specific format on top of a known gap.

3. Public runtime contract without the required consumers. The spec describes the output as "an observation artifact for external consumers", and verify-pack.mjs adds the modules to both the npm required list and the Qoder runtime bundle. ADR-0001: "Versioned public runtime contracts shared by multiple repo surfaces or packaged hosts go to schemas/. Capability-private schemas stay under the owning scripts/<capability>/." ADR-0002 sets the schemas/ activation gate at "Two real consumers, compatibility policy, fixtures, and package-boundary verification." There are currently zero consumers, and policyUse is advisory-only. Either it is capability-private, in which case it should not claim external consumers or enter the pack required list, or it belongs in schemas/ and the gate is not met.

Two factual observations to go with that:

  • Six of the thirteen capability rows are hard-coded constants that are permanently unavailable (FIXED_GAP_REASON_BY_CAPABILITY), and the spec states v1 can never emit evidenceStatus: "complete".
  • docs/specs/fixtures/lc03-no-session-v1.jsonl spends twelve records conveying one bit: the caller declared no session evidence.

Also note git merge-tree reports conflicts against current main in registry.mjs, cli.mjs, and test/fixtures/scripts-refactor-contract/root-help.txt.

What I would keep

The design instinct here is right and I want it preserved: refusing timestamp-proximity inference, and requiring either an exact Episode ref or an explicit no-session declaration. That is precisely where the low tier of commit-session-link is weakest, so the two capabilities are genuinely complementary once they share one vocabulary.

Concretely, my suggestion is to close this PR and reopen the work as the joint alignment spec described above. If you want something landable right now, *.jsonl text eol=lf in .gitattributes is a correct fix on its own and can go in as a separate one-line PR.

@phodal phodal closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Feature request: Task-scoped machine output for external delivery pipelines

3 participants