diff --git a/.gitattributes b/.gitattributes index b50c450..d118650 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,6 +2,7 @@ *.js text eol=lf *.json text eol=lf +*.jsonl text eol=lf *.md text eol=lf *.mjs text eol=lf *.yaml text eol=lf diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a00141..47c8057 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,11 @@ observable behavior and compatibility, not every internal refactor. `kimi-workspace-index-absent` warning. The public npm package now ships seven host metadata roots; the Qoder runtime bundle remains Qoder-specific. +- `better-harness harness experience-trace` adds a read-only Qoder + report-source create/validate v1. Callers explicitly bind task, workspace, + and run keys and select either one Episode or no-session evidence; output is + bounded, privacy-safe canonical JSONL. + - A read-only native Learning Capture review contract can now screen ordinary Task Episodes for repeated exact repair routes, emit a bounded privacy-safe packet, validate evidence-bound `match` or `abstain` decisions, and project diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4eb5035..fb324d2 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -105,6 +105,14 @@ the target journey, contract, evidence, governance, and DX-measurement system. its bounded human-review integrity chain: packet binding, episode/delivery normalization, and review application. Non-facade consumers import its public `index.mjs`; historical flat paths remain compatibility entrypoints. +- `scripts/experience-trace/` owns bounded, read-only Experience Trace JSONL: + its exact wire contract, Qoder report-source allowlist projection, and + create/validate command. It may consume report-source validation only through + that capability's public `index.mjs`; report-source, session-analysis, + Learning Capture, Harness Run Checkpoints, and component snapshots do not + import Experience Trace. Its `index.mjs` is the sole cross-capability + behavioral import surface; the root registry may project only its pure command + manifest so root help does not load runtime behavior. - `scripts/harness-component-snapshot/` owns the versioned, read-only Qoder project component snapshot, validation, bounded diff, and non-authorizing rollback-reference contracts. It composes only allowlisted public diff --git a/docs/specs/2026-08-11-lc03-experience-trace-v1.md b/docs/specs/2026-08-11-lc03-experience-trace-v1.md new file mode 100644 index 0000000..eb5824d --- /dev/null +++ b/docs/specs/2026-08-11-lc03-experience-trace-v1.md @@ -0,0 +1,783 @@ +# Emit a Task-Scoped Experience Trace + +## Traceability + +- Spec ID: `lc03-experience-trace-v1` +- Roadmap: `LC-03` +- Related external evidence: [GitHub Issue #70](https://github.com/QoderAI/better-harness/issues/70) +- AI involvement: Codex (GPT 5.6 Sol) with Terra max subagent review +- Status: Implemented + +## Intent + +Add the first executable LC-03 slice: a read-only command that projects one +explicit Task Episode from an already validated Qoder `report.source.json` into +a reader-safe, versioned JSONL Experience Trace. + +The caller supplies opaque task, workspace, and run keys. Better Harness +projects those keys to domain-separated pseudonymous references, binds the +references to an exact allowlisted source projection, and never +guesses a task from the newest report, session, file modification time, or a +temporal-nearness heuristic. A caller with no applicable native session +evidence can explicitly emit a bound `unavailable` trace instead of allowing an +unrelated local session to look like evidence for the run. + +This slice is an observation artifact for external consumers. It does not make +a gate decision, implement a runtime, persist a checkpoint, or complete every +capability in the full LC-03 roadmap row. + +## Design Inputs and Evidence + +- Issue #70 is related external evidence for the same consumption pain: it asks + for parser-safe output that is bound to one task, workspace, + and run, and that distinguishes complete, partial, and unavailable evidence. + It explicitly keeps deterministic validation and delivery policy outside + Better Harness. It also says that a new public schema is not requested in its + first phase. This LC-03 proposal is therefore independently justified by the + roadmap, does not claim to implement or close #70, and requires normal + maintainer review as a new public contract. +- `scripts/session-analysis/episode-contract.mjs` owns Task Episode grouping, + validation relations, repair candidates, and permission-boundary facts. Its + public Episode object still contains session ids, timestamps, paths, and raw + evidence locators, so it is not a reader-safe trace contract. +- `scripts/harness-analysis/task-loop-source.mjs` already projects Episode paths + to bounded target keys and removes session ids from its Task Episode rows. + The whole report source is still not safe to serialize: unrelated nested + surfaces may retain private debug/session locators, and the report-source + validator does not make every nested Episode field a closed schema. +- `scripts/harness-component-snapshot/` owns the LC-02 snapshot and can validate + a snapshot in isolation. A report source has no comparable workspace or + topology identity, so V1 cannot prove that an arbitrary snapshot and source + belong to the same workspace. +- `docs/specs/2026-08-10-harness-run-checkpoints.md` is a Draft with a closed V1 + artifact and anchor set. Experience Trace must not silently become a new + checkpoint artifact or resume capability. +- The implemented LC-05 specs defer native `repeated-rediscovery` until a + privacy-safe work/read trace exists. This PR establishes a trace boundary but + does not change LC-05 mining. + +## Decision + +Create a new atomic capability under `scripts/experience-trace/`. It consumes a +validated report source through the report-source public index, constructs a +strict allowlist projection, emits a complete JSONL stream on stdout, and can +validate a previously captured stream. + +The root route has maintainer discovery audience and remains a thin dispatcher; +audience filters help/inventory visibility and is not runtime authorization: + +```text +better-harness harness experience-trace create \ + --source \ + --task-key \ + --workspace-key \ + --run-key \ + (--episode-ref episode: | --no-session-evidence) \ + --jsonl + +better-harness harness experience-trace validate --trace +``` + +`create` requires the explicit `--jsonl` stream selector and emits only JSONL +on stdout. It writes no file; callers may capture stdout using their own +process API. `validate` emits one JSON document on +stdout. Both commands complete all validation before emitting success output. + +Both V1 modes require an exact Qoder report source at schema version 3 with a +session observation manifest at schema version 2. Episode mode selects one +Qoder Episode. `--no-session-evidence` is explicit rather than inferred: it +uses the same Qoder source contract but deliberately ignores unrelated retained +Episodes and produces a completely framed stream whose evidence status is +`unavailable`. It is mutually exclusive with `--episode-ref`. Supporting an +external/API-specific source producer requires a later spec. + +## Identity and Binding + +`taskKey`, `workspaceKey`, and `runKey` are caller-owned correlation keys, not +raw task text, paths, repository URLs, session ids, or credentials. Each input +must match `^[A-Za-z0-9][A-Za-z0-9._-]{15,95}$`; the alphabet therefore cannot +contain a path separator, URL scheme, email address, home shortcut, assignment, +or whitespace. Callers should still use randomly generated or +already-pseudonymous identifiers because a stable hash of a guessable business +label remains guessable by dictionary attack. + +Raw keys are never emitted or included in stable errors. The public references +have these exact forms: + +```text +task:sha256:<64 lowercase hexadecimal characters> +workspace:sha256:<64 lowercase hexadecimal characters> +run:sha256:<64 lowercase hexadecimal characters> +``` + +Each reference hashes UTF-8 NFC input with a domain-separated prefix: + +```text +sha256("better-harness:experience-trace:binding:v1\0" + kind + "\0" + key) +``` + +In every digest formula in this spec, `\0` means one NUL byte (`0x00`), never +the two ASCII bytes backslash and zero. `kind` is exactly `task`, `workspace`, +or `run`. Stable hashing is a +pseudonym, not encryption or proof that a key described the correct business +object. Header binding rows therefore declare `provenance: caller-asserted`; +V1 never labels them source-verified. Consumers compare bindings by applying +the same function to their own keys. + +The normative binding vector is: + +| Kind | Raw test key | Required lowercase digest | +| --- | --- | --- | +| task | `task-key-00000001` | `fce82f8f9ff62cd9af044e2e21bfd2a1822d4cc506bf174a285b8fd03fb011e9` | +| workspace | `workspace-key-00000001` | `50c688560321b967e32f410e861cf64cc7af45c817bddbe466aa1f703a29850e` | +| run | `run-key-00000001` | `7522f027d7bb67cbdbb3818345249dc1f4429fe8e484f44c6330083f1292e1d0` | + +For the task row, the complete SHA-256 preimage bytes in hexadecimal are +`6265747465722d6861726e6573733a657870657269656e63652d74726163653a62696e64696e673a7631007461736b007461736b2d6b65792d3030303030303031`. +The two `00` byte pairs are the required NUL separators. + +The source Episode id remains a source-scoped opaque `episodeRef`. It is not a +global task id: the current Episode owner may derive it from a session id plus +ordinal or from an explicit task key. Trace identity therefore binds: + +```text +schema version ++ producer platform ++ pseudonymous taskRef ++ pseudonymous workspaceRef ++ pseudonymous runRef ++ sourceProjectionDigest ++ selected episodeRef or explicit no-session marker +``` + +`traceId` is a SHA-256 identifier over those canonical fields. Equal raw +session/task ids in another provider, workspace, or run cannot collide unless +all caller assertions and the projected evidence are also equal. This is +namespace separation, not verification that the caller selected the right +Episode for its business task. + +`sourceProjectionDigest` covers only the allowlisted reader-safe facts that +the trace uses. It does not hash the complete report source, because private +or unrelated source fields must not influence a public pseudonym. + +## Source Projection + +After bounded JSON parsing and an iterative structural preflight, the projector +checks the following safe top-level source markers, then runs +`validateHarnessReportSource()` as an integrity prerequisite. That validator is +not proof that the whole object is publishable. V1 requires every marker +exactly before the broader validator: + +| Source field | Required value | +| --- | --- | +| `schemaVersion` | `3` exactly, not merely `>= 3` | +| `kind` | `harness-report-source` | +| `manifest.schemaVersion` | `2` exactly | +| `manifest.kind` | `session-observation-manifest` | +| `manifest.scope.platform` | `qoder` | +| `manifest.scope.workspaceScope` | `workspace` | + +Marker failure classification is exact: + +| Marker failure | Code | +| --- | --- | +| malformed JSON; root/manifest/scope is missing, array, null, or not an object | `INVALID_REPORT_SOURCE` | +| source or manifest `schemaVersion` is missing or not an integer | `INVALID_REPORT_SOURCE` | +| source or manifest `kind` is missing, non-string, or not its required value | `INVALID_REPORT_SOURCE` | +| an integer source or manifest schema version differs from the required value | `UNSUPPORTED_TRACE_SOURCE_VERSION` | +| platform or workspace scope is missing or non-string | `INVALID_REPORT_SOURCE` | +| a string platform or workspace scope differs from the required value | `UNSUPPORTED_TRACE_PLATFORM` | + +Rows are evaluated top to bottom and the first matching row is the only emitted +failure. Thus malformed structure outranks every later marker, invalid kind +outranks unsupported integer version, and unsupported integer version outranks +unsupported platform/scope. Combined failures never produce multiple codes. + +The preflight walks iteratively, without invoking recursive product validators, +and permits at most depth 64 and 100,000 total JSON values. Root depth is zero +and each property/element edge adds one. The root counts as one value; every +object-property value and array element adds one; property-name keys +do not count. Containers therefore count as the value by which they were +reached, not once again for their children. Exceeding either structural +budget is `INVALID_REPORT_SOURCE`. Only after this check may the existing broad +validator run; any thrown exception is caught and normalized to the same stable +code without its message or stack. + +Supporting a later report/manifest schema is a contract change, not an +automatic forward-compatibility promise. Both Episode and no-session mode use +this same compatibility rule. An API-only producer is deferred; the V1 +no-session mode models its evidence absence using a validated Qoder source and +does not pretend an API transcript was inspected. + +After compatibility checks, the projector constructs this exact internal +object from scratch. It never spreads a source object: + +```text +projection = { + schemaVersion: 1, + producer: { + platform: "qoder", + reportSourceSchemaVersion: 3, + manifestSchemaVersion: 2 + }, + selection: { + sourceFingerprint, + strategy, + eligibleCount, + analyzedCount, + sampled, + representative, + confidence, + warningCodes + }, + episode: null | { + episodeRef, + sessionCount, + continuation, + startBoundary, + toolCallCount, + changeSets: [{ eventCount, firstOrdinal, lastOrdinal, targetKeys }], + validationSets: [{ category, status, sourceOrdinal, + checkIdentity, targetKeys }], + permissionBoundary: null | { + prompted, denied, escalated, protectedActions + }, + closureStatus, + repairStatus + }, + absenceReason: null | "caller-declared-no-session-evidence" +} +``` + +Every projected value has one source owner. A missing required object, array, +or scalar; a wrong type; or a value outside the rule table below fails with +`INVALID_REPORT_SOURCE`, even when the broader report-source validator accepted +it. Nullable rows are absent only under the stated condition. + +| Projection path | Exact report-source path or construction | +| --- | --- | +| `producer.*` | constants from the six compatibility markers above | +| `selection.sourceFingerprint` | `manifest.sources.fingerprint` | +| `selection.strategy` through `selection.confidence` | same-named fields under `manifest.selection` | +| `selection.warningCodes` | `manifest.warningCodes`, validated before canonical set normalization | +| `episode` | the unique `taskEpisodes[]` row whose `id` exactly equals `--episode-ref`; `null` only in no-session mode | +| `episode.episodeRef` | selected row `id` | +| `sessionCount`, `continuation`, `startBoundary`, `toolCallCount` | selected row `sessionCount`, `continuation`, `startBoundary`, and `toolCalls` | +| `changeSets[]` | selected row `changeSets[]` in source-array order | +| each change-set scalar | same-named `eventCount`, `firstOrdinal`, and `lastOrdinal` source field | +| each change-set `targetKeys` | its source row `targetKeys`, validated before canonical set normalization | +| `validationSets[]` | selected row `validationSets[]`, sorted by numeric `ordinal`; duplicate ordinals fail | +| each validation scalar | source `category`, `status`, `ordinal -> sourceOrdinal`, `checkIdentity`, and `targetKeys` | +| `permissionBoundary` | selected row `permissionSummary`, omitting only `evidenceRefs`; `null` only when that object is absent | +| `closureStatus` / `repairStatus` | selected row `closure.status` / `repair.status` | +| `absenceReason` | `null` in Episode mode; constant `caller-declared-no-session-evidence` in no-session mode | + +No fallback default is applied. In particular, missing `toolCalls`, status, +count, or array fields are invalid rather than silently becoming zero, an empty +array, or `unobserved`. + +The exact value rules are: + +| Field | V1 rule | +| --- | --- | +| `sourceFingerprint` | exactly 16 lowercase hexadecimal characters | +| `strategy` | `stratified`, `all-eligible`, or `latest-n` | +| selection counts | `analyzedCount <= eligibleCount` in addition to the shared integer bound | +| selection booleans | JSON booleans; `sampled === (analyzedCount < eligibleCount)` and `representative === (!sampled && strategy !== "latest-n")` | +| `confidence` | `Low` when either count is zero or strategy is `latest-n`; otherwise `Medium` when sampled and `High` when not sampled | +| `warningCodes` | source array at most 32; every item is one of `missing-required-root`, `missing-optional-root`, `disabled-source-root`, `partial-secret-scan-coverage`, or `invalid-prior-learning-capture-report`; output is code-unit-sorted unique | +| every count/ordinal | safe non-negative integer at most 1,000,000 | +| `episodeRef` | `^episode:[a-f0-9]{12,64}$` and exact match to one retained source Episode | +| `sessionCount` | integer 1-1,000,000; a value over one requires `continuation: explicit` | +| `continuation` | `explicit` or `session-bounded` | +| `startBoundary` | `session-start`, `explicit-task-key`, `explicit-boundary`, `idle-gap`, `progress-handoff`, or `first-retained-boundary` | +| `changeSets` | source order, at most 16 rows; `firstOrdinal <= lastOrdinal`; each raw target-key array has at most 12 items and output is code-unit-sorted unique | +| `targetKeys` | exactly 20 lowercase hexadecimal characters | +| `validationSets` | raw source array at most 32 rows with unique source ordinals; output is numeric-ordinal-sorted; each raw target-key array has at most 12 items and output is code-unit-sorted unique | +| validation `category` | `npm test`, `pnpm test`, `yarn test`, `node --test`, `vitest`, `jest`, `pytest`, `go test`, `cargo test`, `agent-lint`, `typecheck`, `lint`, or `git diff --check` | +| validation `status` | `passed`, `failed`, or `observed` | +| `checkIdentity` | `^check:[a-f0-9]{24}$` | +| permission counts | each 0-1,000,000; prompted/denied/escalated do not exceed protected actions; the object is present only when protected actions is positive | +| `closureStatus` | `not-applicable`, `unobserved`, `closed`, or `observed-without-pass` | +| `repairStatus` | `not-applicable`, `unobserved`, `review-required`, or `repaired-and-passed` | + +An allowlisted scalar or bounded row that violates those rules fails closed +rather than being cleaned, truncated, or replaced. Duplicate or unsorted input +is explicitly accepted for the set-like `warningCodes` and `targetKeys` arrays: +first validate the raw array length and every member, then deduplicate, then +sort with the code-unit comparator. Unsorted `validationSets` are also accepted +after every row and the raw bound validate; duplicate numeric ordinals fail, +and unique rows sort numerically by ordinal. No other array is reordered or +deduplicated. Source +evidence ids, adapter labels, +timestamps, elapsed duration, task routes, lifecycle/learning signals, and +session/invocation identities are not part of the projection. + +When an Episode exists, `--episode-ref` is always required, even if the source +contains only one Episode. An absent or unknown Episode ref fails closed. +`--no-session-evidence` sets `episode: null` and the exact absence reason above; +it never falls through to an unrelated retained Episode. + +## JSONL Contract + +Every line is one exact-schema object. Unknown or missing fields fail. The +common fields are `kind`, `schemaVersion: 1`, zero-based `recordOrdinal`, and +one `traceId`. Ordinals increase by exactly one. There is exactly one header, +one or more events, and one terminal; no content may follow the terminal. + +### Canonical bytes and hashes + +Canonical JSON recursively sorts object keys by ascending UTF-16 code-unit +order, preserves array order, normalizes every string to Unicode NFC, permits +only JSON booleans/null, strings, and safe integers, and serializes with compact +`JSON.stringify` semantics. Each record is that UTF-8 JSON with no BOM followed +by one LF byte (`0x0a`), including the terminal. CRLF, insignificant whitespace, +non-canonical key order, duplicate JSON keys, unsafe integers, floats, negative +zero, lone surrogates, or a missing final LF are invalid. + +The exact domain-separated digests are: + +```text +sourceProjectionDigest = "sha256:" + hex(sha256(utf8( + "better-harness:experience-trace:source-projection:v1\0" + + canonicalJson(projection)))) + +traceId = "trace:sha256:" + hex(sha256(utf8( + "better-harness:experience-trace:identity:v1\0" + + canonicalJson({ + schemaVersion: 1, + producerPlatform: "qoder", + taskRef, workspaceRef, runRef, + sourceProjectionDigest, + episodeRef: selectedEpisodeRefOrNull, + absenceReason: explicitAbsenceReasonOrNull + })))) + +traceDigest = "sha256:" + hex(sha256(utf8( + "better-harness:experience-trace:stream:v1\0" + + canonicalHeaderLineWithLF + + everyCanonicalEventLineWithLF))) +``` + +The terminal is not included in `traceDigest`. A validator reconstructs the +safe projection from the header/events, recomputes all three digests, and +requires exact equality. This detects accidental or non-recomputed mutation; +it is not a signature and cannot authenticate a malicious producer that +recomputes the whole stream. + +The normative no-session golden vector uses the three binding keys from the +Identity and Binding table, source fingerprint `0123456789abcdef`, selection +`all-eligible`, zero eligible/analyzed, `sampled: false`, +`representative: true`, `confidence: Low`, no warnings, and the fixed ten gap +events in coverage order. It must produce: + +| Value | Required result | +| --- | --- | +| `sourceProjectionDigest` | `sha256:f46c5aaea639376da6fca7bfa9df215ee2a22bdad509761912a705d57d8eb9c9` | +| `traceId` | `trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f` | +| `traceDigest` | `sha256:e568040d4ce087a63abb593e760ea4b94b140a8da6421dad7b69ab583426956e` | +| counts | `eventCount: 10`, `recordCount: 12`, terminal ordinal `11` | + +The complete byte stream is checked in beside this spec as the +[normative no-session fixture](fixtures/lc03-no-session-v1.jsonl). +Implementation tests must consume that exact file and add a second Episode +fixture. Changing a byte-affecting rule requires a schema revision or an +explicitly reviewed compatibility decision, not silently updating the vectors. + +### Header record + +The header has exactly this shape: + +```text +{ + kind: "better-harness.experience-trace.header", + schemaVersion: 1, + recordOrdinal: 0, + traceId, + binding: { + task: { ref: taskRef, provenance: "caller-asserted" }, + workspace: { ref: workspaceRef, provenance: "caller-asserted" }, + run: { ref: runRef, provenance: "caller-asserted" }, + episode: { + ref: episodeRefOrNull, + provenance: "source-projected" | "unavailable", + reasonCode: "explicit-episode-selection" + | "caller-declared-no-session-evidence" + } + }, + source: { + projectionDigest: sourceProjectionDigest, + reportSourceSchemaVersion: 3, + manifestSchemaVersion: 2 + }, + producer: { + kind: "qoder-report-source-projection", + platform: "qoder" + }, + selection: { + sourceFingerprint, strategy, eligibleCount, analyzedCount, + sampled, representative, confidence, warningCodes + } +} +``` + +`episode.ref` is a string with `source-projected` provenance in Episode mode; +it is `null` with `unavailable` provenance in no-session mode. No field implies +that caller bindings were checked against the report workspace. + +### Event records + +Every middle record has exactly: + +```text +{ + kind: "better-harness.experience-trace.event", + schemaVersion: 1, + recordOrdinal, + traceId, + eventType, + availability, + evidenceRef, + payload +} +``` + +The event-specific contract is: + +| Event type | Availability / evidence ref | Exact payload | +| --- | --- | --- | +| `task-episode` | `observed` / `source:episode` | `{ episodeRef, sessionCount, continuation, startBoundary }` | +| `tool-observation` | `derived` / `source:tool-summary` | `{ toolCallCount, changeSets: [{ changeRef, eventCount, firstOrdinal, lastOrdinal, targetKeys }] }`; `changeRef` is `source:change:<1-based-index>` | +| `validation-observation` | `observed` / `source:validation:<1-based-position-after-sourceOrdinal-sort>` | `{ validationRef, category, status, sourceOrdinal, checkIdentity, targetKeys }`; `validationRef` equals the evidence ref and is based on normalized output position, never original array position | +| `permission-boundary` | `derived` / `source:permission-boundary` | `{ prompted, denied, escalated, protectedActions }` | +| `episode-observation-ended` | `derived` / `source:episode-end` | `{ closureStatus, repairStatus }` | +| `capability-gap` | `unavailable` / `source:gap:` | `{ capability, reasonCode }` | + +Episode event order is task Episode, tool observation, validation observations +by ascending `sourceOrdinal` (ties are invalid), optional permission boundary, +Episode end, and capability gaps in terminal coverage order. No-session order is +only capability gaps in terminal coverage order. Each unavailable coverage row +has exactly one matching gap event; no non-unavailable row has one. + +### Terminal record + +The terminal has exactly: + +```text +{ + kind: "better-harness.experience-trace.terminal", + schemaVersion: 1, + recordOrdinal, + traceId, + streamStatus: "complete", + evidenceStatus: "partial" | "unavailable", + coverage, + eventCount, + recordCount, + policyUse: "advisory-only", + traceDigest +} +``` + +`eventCount` counts middle records only; `recordCount === eventCount + 2`; the +terminal ordinal is `recordCount - 1`. `streamStatus` says only that framing is +complete. `evidenceStatus` is `unavailable` when task-Episode coverage is +unavailable and `partial` otherwise. V1 can never emit evidence status +`complete` because bindings remain caller-asserted and six required runtime +capabilities remain unavailable; a future complete producer requires a schema +revision. Neither status is a pass, approval, gate, or delivery decision. + +## Capability Coverage + +The terminal always contains these rows in this order: + +1. `task-binding` +2. `workspace-binding` +3. `run-binding` +4. `task-episode` +5. `tool-observation` +6. `validation-observation` +7. `permission-boundary` +8. `human-approval` +9. `subagent-lineage` +10. `worktree-lineage` +11. `interruption-resume` +12. `runtime-stop` +13. `component-snapshot` + +Each exact row is `{ capability, availability, provenance, reasonCode }`. +`availability` is `observed | derived | unavailable`; `provenance` is +`caller-asserted | source-projected | unavailable`. The deterministic matrix +is: + +| Capabilities | Episode mode | No-session mode | +| --- | --- | --- | +| task/workspace/run binding | `derived`, `caller-asserted`, `explicit-caller-binding` | same | +| task Episode | `observed`, `source-projected`, `explicit-episode-selection` | `unavailable`, `unavailable`, `caller-declared-no-session-evidence` | +| tool observation | `derived`, `source-projected`, `source-aggregate` | `unavailable`, `unavailable`, `caller-declared-no-session-evidence` | +| validation observation | `observed`, `source-projected`, `source-observation` when rows exist; otherwise `unavailable`, `unavailable`, `not-retained-by-source` | `unavailable`, `unavailable`, `caller-declared-no-session-evidence` | +| permission boundary | `derived`, `source-projected`, `source-permission-summary` when present; otherwise `unavailable`, `unavailable`, `not-retained-by-source` | `unavailable`, `unavailable`, `caller-declared-no-session-evidence` | +| six fixed gaps below | `unavailable`, `unavailable`, capability-specific reason | same | + +The six fixed gap reasons are `human-approval-not-observed`, +`subagent-parent-edge-not-observed`, `worktree-lineage-not-observed`, +`interruption-resume-not-observed`, `runtime-stop-not-observed`, and +`component-snapshot-not-bound`. Those reason codes map one-to-one, in order, to +human approval, subagent lineage, worktree lineage, interruption/resume, +runtime stop, and component snapshot. The complete closed reason-code set is +those six plus `explicit-caller-binding`, `explicit-episode-selection`, +`source-aggregate`, `source-observation`, `source-permission-summary`, +`caller-declared-no-session-evidence`, and `not-retained-by-source`. + +V1 always reports these gaps rather than substituting proxies: + +- permission allowed/denied is not human approval; +- `isSubagent` without a parent edge is not lineage; +- an idle gap is not interruption/resume; +- Episode validation closure is not runtime stop; +- an unbound LC-02 snapshot is not component state for this run. + +## Privacy, Bounds, and Error Contract + +No record or stable error may contain raw prompts, commands, transcript text, +session ids, invocation ids, timestamps, credentials, repository URLs, +absolute Windows/POSIX/UNC paths, home paths, or raw evidence locators. + +Create reads at most 16 MiB of source JSON and applies the depth-64/100,000-value +iterative preflight before any recursive product validator. Both commands use one shared trace +bound: at most 1 MiB total serialized JSONL, 256 records, 65,536 UTF-8 bytes per +line, and 64 events. Projection arrays use the smaller limits above. Create +serializes into a bounded buffer, runs the same byte-level validator used by +`validate`, and only then writes stdout. Therefore every successful create +stream must be accepted byte-for-byte by validate. Bounds are checked before +success stdout is written. + +After each trace line parses as JSON but before canonicalization or recursive +schema validation, validate applies the same iterative structural algorithm: +root depth zero, maximum depth 64, and at most 100,000 total JSON values summed +across all records in the stream. Structural overflow or any caught parser, +canonicalizer, or schema traversal exception maps to +`INVALID_EXPERIENCE_TRACE`; it never exposes an exception message or stack. + +Help writes usage text to stdout, nothing to stderr, exits 0, and performs no +source, trace, workspace, home, Git, or network read. Successful create writes +only the complete canonical JSONL stream to stdout and nothing to stderr. +Successful validate writes this one canonical JSON document plus LF to stdout +and nothing to stderr: + +```text +{ + kind: "better-harness.experience-trace.validation", + schemaVersion: 1, + valid: true, + traceId, + streamStatus, + evidenceStatus, + recordCount, + eventCount, + traceDigest +} +``` + +`validate --trace` checks the bytes, closed schemas, deterministic projection +reconstruction, identity, counts, ordering, and digests. It does not read the +original report source and therefore does not authenticate the source or the +caller assertions. + +Only the exact argv forms `--help`, `-h`, `create --help`, `create -h`, +`validate --help`, and `validate -h` are help. They take priority over runtime +work and perform zero reads. Any help token combined with another token is not +help and follows strict argument validation. + +On every usage, read, source, bounds, or validation failure the leaf buffers and +discards all candidate output, writes nothing to stdout, and writes exactly one +ASCII line `: \n` to stderr. It never exposes a path, +argument value, validator detail, stack, or nested `errors`. The exact V1 +mapping and precedence are: + +| Condition | Code | Exit | Stable message | +| --- | --- | --- | --- | +| unknown phase/flag/positional; missing or duplicate option value; missing `--jsonl` on create; help plus any extra token; both Episode modes | `INVALID_USAGE` | 64 | `invalid experience-trace arguments` | +| create has neither Episode mode after otherwise valid parsing | `MISSING_EPISODE_SELECTION` | 64 | `select exactly one episode mode` | +| a task/workspace/run key fails its exact syntax | `INVALID_TRACE_BINDING` | 1 | `trace binding key is invalid` | +| source file I/O fails | `SOURCE_READ_FAILED` | 1 | `unable to read report source` | +| source exceeds 16 MiB or the constructed trace exceeds a shared trace bound | `TRACE_BOUNDS_EXCEEDED` | 1 | `experience trace bounds exceeded` | +| source JSON parse, broad validation, or allowlist projection fails | `INVALID_REPORT_SOURCE` | 1 | `report source is invalid` | +| source/manifest version marker is unsupported | `UNSUPPORTED_TRACE_SOURCE_VERSION` | 1 | `report source version is unsupported` | +| source platform or workspace scope is unsupported | `UNSUPPORTED_TRACE_PLATFORM` | 1 | `report source platform is unsupported` | +| `--episode-ref` is malformed, absent, or non-unique in the retained Episode rows | `UNKNOWN_EPISODE_REF` | 1 | `selected episode is unavailable` | +| trace file I/O fails | `TRACE_READ_FAILED` | 1 | `unable to read experience trace` | +| trace exceeds a shared trace bound | `TRACE_BOUNDS_EXCEEDED` | 1 | `experience trace bounds exceeded` | +| trace bytes, parse, schema, reconstruction, ordering, counts, or digests fail | `INVALID_EXPERIENCE_TRACE` | 1 | `experience trace is invalid` | + +Argument shape is checked first, then binding syntax, source/trace I/O and byte +bounds, source version/platform markers, broad source validation/projection, or +trace validation as applicable. Thus a future source version reports the +version code even if that version would fail the current broad validator. File +read failures never echo the private filename. Parse/source failures never echo +parser or validator messages. Root dispatch treats Experience Trace as an +owner-validated-help command and preserves the leaf argv, stdout, stderr, and +exit code, including invalid trailing-help cases. + +The capability-owned command declaration is `experience-trace.v1`, audience +`maintainer`, with phases `create` and `validate`. It is `read-only`: create +reads only the explicitly named source file; validate reads only the explicitly +named trace file; neither reads the current workspace, Git, host home, user +state, environment-derived evidence roots, or network. Neither phase writes or +persists an artifact, and Better Harness retains nothing. Raw caller keys exist +only in argv/process memory for reference derivation. + +## Ownership and Planned Files + +- `scripts/experience-trace/contract.mjs` owns exact records, canonical JSON, + identity/digests, bounds, parsing, validation, and JSONL serialization. +- `scripts/experience-trace/project-source.mjs` owns the strict report-source + allowlist projection and deterministic record construction. +- `scripts/experience-trace/index.mjs` is the only cross-capability behavioral + import surface. +- `scripts/experience-trace/cli.mjs` owns create/validate parsing, stable errors, + and stdout behavior. +- `scripts/experience-trace/command-manifest.mjs` owns pure route metadata, + audience, options, effects, examples, and diagnostics used by leaf help and + indexed by the root registry without loading runtime behavior. +- `scripts/better-harness-cli/registry.mjs` indexes that metadata. +- `scripts/better-harness-cli/cli.mjs` forwards Experience Trace help argv so + the leaf, not the facade, owns strict help validation. +- `docs/ARCHITECTURE.md` records the one-way dependency: Experience Trace may + consume report-source public validation, but report/session/LC-05/Checkpoint + owners do not import Experience Trace. +- Tests own golden records, adversarial privacy fixtures, root dispatch, help + side-effect checks, and package verification expectations. + +The implementation must not modify `task-loop-source.mjs`, session provider +adapters, LC-05 candidate/review owners, LC-02 snapshot owners, or the +Checkpoint Draft. This minimizes collision with open PR #72 and preserves the +existing report chain. + +## Acceptance Scenarios + +- **ET-AC-1 (framed deterministic JSONL):** Given one validated Qoder report + source, explicit bindings, and an exact Episode ref, repeated create calls + emit byte-identical JSONL with one header, canonical event order, and one + terminal whose counts and digest validate. +- **ET-AC-2 (explicit task/run/workspace binding):** The trace id and header bind + safe `taskRef`, `workspaceRef`, `runRef`, producer platform, safe source + projection, and Episode ref. Reusing a source Episode id with different caller + workspace/run keys produces a different trace id; deliberately reusing all + caller keys remains a caller assertion, not a verified source binding. No + newest/mtime/temporal fallback exists. +- **ET-AC-3 (explicit Episode or absence):** Episode mode requires an exact + source Episode ref. `--no-session-evidence` is mutually exclusive and emits a + completely framed `unavailable` trace even if unrelated source Episodes + exist. Missing or invented selection fails before stdout. +- **ET-AC-4 (fact and capability honesty):** Tool, validation, permission, and + Episode-end rows preserve only allowlisted observed/derived facts. Human + approval, subagent/worktree lineage, interruption/resume, runtime stop, and + component snapshot remain explicit gaps with closed reasons. +- **ET-AC-5 (strict privacy projection):** Private sentinels injected into raw + prompt/command/transcript/session/invocation ids, unknown Episode fields, + evidence paths, absolute Windows/POSIX/UNC/home paths, URLs, emails, and + secret-like values never appear in trace stdout or stable errors. Unknown + fields cannot enter the projection by object spread. +- **ET-AC-6 (coverage semantics):** Stream validity and evidence availability + are independent. A terminal-complete empty stream is `unavailable`; the + initial Qoder Episode producer is `partial`; neither produces a pass or gate + decision. +- **ET-AC-7 (fail-closed validation):** Unsupported platform, invalid source, + unsafe binding, oversize input, unknown record/type/field/enum, reordered or + duplicate ordinal, missing/extra terminal content, stale counts, and digest + tampering fail with stable no-path errors. +- **ET-AC-8 (read-only public route):** Root CLI help is zero-read; create and + validate write no workspace, host-home, user-state, or Checkpoint data; + machine stdout is parser-safe and root dispatch preserves it byte-for-byte. +- **ET-AC-9 (compatibility and ownership):** Existing report-source, Task + Episode, LC-02, LC-05, and long-session trace behavior remains unchanged. + Cross-capability imports use public indexes and the packaged artifact contains + the complete Experience Trace owner. +- **ET-AC-10 (external consumer boundary):** A fixture representing an API + Maker task with no applicable native Qoder Episode uses the explicit + no-session mode against a valid Qoder source and yields unavailable coverage + while retaining caller-asserted task/workspace/run bindings. It is not an API + transcript producer. Output exposes the facts needed for an external consumer + to choose advisory or human-review handling, but Better Harness does not + define or execute its delivery gate. This PR is related evidence for Issue + #70 and does not claim to close its findings or delivery-policy request. + +## Non-goals + +- Do not complete the full cross-host LC-03 roadmap row or mark it done. +- Do not add a host adapter or change Qoder/session discovery and selection. +- Do not emit raw per-tool transcripts, arguments, results, prompts, commands, + content, paths, timestamps, session ids, or native stop text. +- Do not infer or record real human approval, subagent/worktree lineage, + interruption/resume, runtime stop, or side-effect state. +- Do not add OTLP export. +- Do not create, validate, store, list, or resume a Harness Checkpoint, and do + not add Experience Trace to the Checkpoint Draft's closed artifact set. +- Do not create, bind, diff, resolve, or authorize an LC-02 component snapshot. +- Do not add LC-05 `repeated-rediscovery` or alter candidate/review contracts. +- Do not add a scheduler, durable runtime, gate, apply, retry, budget, recovery, + or release decision. +- Do not change report-source or findings schemas in this slice. + +## Plan and Tasks + +1. Implement the strict trace contract, canonical identity/digest, parser, + serializer, bounds, and golden vectors (ET-AC-1, ET-AC-2, ET-AC-6, ET-AC-7). +2. Implement the allowlist report-source projector and explicit + Episode/no-session modes (ET-AC-3, ET-AC-4, ET-AC-5, ET-AC-10). +3. Add the create/validate CLI, stable errors, and maintainer root registry leaf + with zero-read help behavior (ET-AC-7, ET-AC-8). +4. Add architecture ownership and package/dispatch contract updates without + touching adjacent capability owners (ET-AC-9). +5. Run focused, cross-module, documentation, package, and full regression + checks, then perform Review Readiness and an independent adversarial review. + +## Test and Review Evidence + +The implementation was verified locally with the following evidence; no CI +status is claimed: + +| Lane | Command | Result | +| --- | --- | --- | +| Focused contract, projection, CLI, root-dispatch, and frozen-output tests | `node --test test/experience-trace-contract.test.mjs test/experience-trace.test.mjs test/experience-trace-cli.test.mjs test/better-harness-cli.test.mjs test/scripts-refactor-contract.test.mjs` | 82 tests, 82 passed, 0 failed | +| Cross-module compatibility | `node --test test/harness-report-source.test.mjs test/task-loop-source.test.mjs test/session-episode-contract.test.mjs` | 46 tests, 46 passed, 0 failed | +| Documentation routing | `node scripts/doc-link-graph/cli.mjs skills/better-harness`, then `node --test test/doc-link-graph.test.mjs` | graph regenerated; 6 tests, 6 passed, 0 failed | +| Package and runtime bundle | `npm run pack:verify` | passed; npm package contained 473 files and the runtime bundle contained 495 entries | +| Full regression | `npm test` | 1,323 tests total; 1,322 passed, 0 failed, 1 skipped | +| Diff hygiene | `git diff --check` | passed | + +The final upstream check observed `origin/main` at `81440ba` and this branch +zero commits behind it. Three independent Terra Max reviewers received the +same final adversarial-review prompt; all three reported `verdict=pass` and +`p1_p2_clear=true`. + +Earlier adversarial rounds found and closed an Episode header/event reference +mismatch, pre-parse high-line-count denial-of-service exposure, inconsistent +shared-bound error codes, the missing versioned Episode golden fixture, +invalid-UTF-8 versus bound-precedence drift, and an unterminated trailing-line +counting off-by-one. The final replay covered both golden fixtures, digest and +identity tampering, total/line/record/event bounds, high-line-count input, +invalid UTF-8 combinations, missing final LF, nonzero-offset `Uint8Array` +views, and direct/root CLI output channels without finding another P1 or P2. + +External API Maker execution and native Qoder runtime proof were not performed. +ET-AC-10 is evidenced by the versioned no-session fixture and its explicit +unavailable-coverage contract, not by an external API transcript or native +runtime observation. GitHub Issue #70 remains related external evidence; this +implementation does not claim to close it. + +## Risks + +- A caller can bind the wrong Episode intentionally. V1 proves that the caller + made an explicit, digest-bound selection; it does not claim to know the + caller's business task semantics. The output labels this as caller binding. +- A valid report source may contain unrelated private nested fields. The + projector never serializes or hashes the full source and rejects unsafe + allowlisted values. +- Open PR #38 may overlap root registry/help fixtures, and Draft PR #72 overlaps + broader session/report owners. Rebase from the latest main before final + validation and keep implementation inside the new owner. +- A future Checkpoint or LC-06 runtime may reference Experience Trace. That + requires a separate reviewed contract and must not reinterpret V1 + `unavailable` capability rows as observed history. diff --git a/docs/specs/fixtures/lc03-episode-v1.jsonl b/docs/specs/fixtures/lc03-episode-v1.jsonl new file mode 100644 index 0000000..6ce6108 --- /dev/null +++ b/docs/specs/fixtures/lc03-episode-v1.jsonl @@ -0,0 +1,13 @@ +{"binding":{"episode":{"provenance":"source-projected","reasonCode":"explicit-episode-selection","ref":"episode:abcdefabcdef"},"run":{"provenance":"caller-asserted","ref":"run:sha256:341d2e8ffe91f814b597ef41304711ceac5285aac6d8ff60f23fffcdb292619d"},"task":{"provenance":"caller-asserted","ref":"task:sha256:cf12e4ef94c4409b6a3cead516ae0955167d167da0a75c8e736e8eb2ebad05b4"},"workspace":{"provenance":"caller-asserted","ref":"workspace:sha256:635affe27510691386923ddaeace0c916a348b6664cd31e2b4dc6a479e95ba0e"}},"kind":"better-harness.experience-trace.header","producer":{"kind":"qoder-report-source-projection","platform":"qoder"},"recordOrdinal":0,"schemaVersion":1,"selection":{"analyzedCount":2,"confidence":"Medium","eligibleCount":3,"representative":false,"sampled":true,"sourceFingerprint":"fedcba9876543210","strategy":"stratified","warningCodes":["disabled-source-root","missing-optional-root"]},"source":{"manifestSchemaVersion":2,"projectionDigest":"sha256:670e6779abb62b2dae18d2f5081a11b102bbef3a9baa0799b1828d0666b4ff8b","reportSourceSchemaVersion":3},"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"observed","eventType":"task-episode","evidenceRef":"source:episode","kind":"better-harness.experience-trace.event","payload":{"continuation":"session-bounded","episodeRef":"episode:abcdefabcdef","sessionCount":1,"startBoundary":"session-start"},"recordOrdinal":1,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"derived","eventType":"tool-observation","evidenceRef":"source:tool-summary","kind":"better-harness.experience-trace.event","payload":{"changeSets":[{"changeRef":"source:change:1","eventCount":2,"firstOrdinal":3,"lastOrdinal":4,"targetKeys":["11111111111111111111","22222222222222222222"]}],"toolCallCount":5},"recordOrdinal":2,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"observed","eventType":"validation-observation","evidenceRef":"source:validation:1","kind":"better-harness.experience-trace.event","payload":{"category":"node --test","checkIdentity":"check:aaaaaaaaaaaaaaaaaaaaaaaa","sourceOrdinal":7,"status":"passed","targetKeys":["33333333333333333333"],"validationRef":"source:validation:1"},"recordOrdinal":3,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"derived","eventType":"permission-boundary","evidenceRef":"source:permission-boundary","kind":"better-harness.experience-trace.event","payload":{"denied":0,"escalated":0,"prompted":1,"protectedActions":1},"recordOrdinal":4,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"derived","eventType":"episode-observation-ended","evidenceRef":"source:episode-end","kind":"better-harness.experience-trace.event","payload":{"closureStatus":"closed","repairStatus":"repaired-and-passed"},"recordOrdinal":5,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:human-approval","kind":"better-harness.experience-trace.event","payload":{"capability":"human-approval","reasonCode":"human-approval-not-observed"},"recordOrdinal":6,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:subagent-lineage","kind":"better-harness.experience-trace.event","payload":{"capability":"subagent-lineage","reasonCode":"subagent-parent-edge-not-observed"},"recordOrdinal":7,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:worktree-lineage","kind":"better-harness.experience-trace.event","payload":{"capability":"worktree-lineage","reasonCode":"worktree-lineage-not-observed"},"recordOrdinal":8,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:interruption-resume","kind":"better-harness.experience-trace.event","payload":{"capability":"interruption-resume","reasonCode":"interruption-resume-not-observed"},"recordOrdinal":9,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:runtime-stop","kind":"better-harness.experience-trace.event","payload":{"capability":"runtime-stop","reasonCode":"runtime-stop-not-observed"},"recordOrdinal":10,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:component-snapshot","kind":"better-harness.experience-trace.event","payload":{"capability":"component-snapshot","reasonCode":"component-snapshot-not-bound"},"recordOrdinal":11,"schemaVersion":1,"traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} +{"coverage":[{"availability":"derived","capability":"task-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"derived","capability":"workspace-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"derived","capability":"run-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"observed","capability":"task-episode","provenance":"source-projected","reasonCode":"explicit-episode-selection"},{"availability":"derived","capability":"tool-observation","provenance":"source-projected","reasonCode":"source-aggregate"},{"availability":"observed","capability":"validation-observation","provenance":"source-projected","reasonCode":"source-observation"},{"availability":"derived","capability":"permission-boundary","provenance":"source-projected","reasonCode":"source-permission-summary"},{"availability":"unavailable","capability":"human-approval","provenance":"unavailable","reasonCode":"human-approval-not-observed"},{"availability":"unavailable","capability":"subagent-lineage","provenance":"unavailable","reasonCode":"subagent-parent-edge-not-observed"},{"availability":"unavailable","capability":"worktree-lineage","provenance":"unavailable","reasonCode":"worktree-lineage-not-observed"},{"availability":"unavailable","capability":"interruption-resume","provenance":"unavailable","reasonCode":"interruption-resume-not-observed"},{"availability":"unavailable","capability":"runtime-stop","provenance":"unavailable","reasonCode":"runtime-stop-not-observed"},{"availability":"unavailable","capability":"component-snapshot","provenance":"unavailable","reasonCode":"component-snapshot-not-bound"}],"eventCount":11,"evidenceStatus":"partial","kind":"better-harness.experience-trace.terminal","policyUse":"advisory-only","recordCount":13,"recordOrdinal":12,"schemaVersion":1,"streamStatus":"complete","traceDigest":"sha256:f15892bcafa1ad3ac34586a0bdae412d73e995f84bd72ebe340b5470cf5ae5f1","traceId":"trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"} diff --git a/docs/specs/fixtures/lc03-no-session-v1.jsonl b/docs/specs/fixtures/lc03-no-session-v1.jsonl new file mode 100644 index 0000000..54edf6d --- /dev/null +++ b/docs/specs/fixtures/lc03-no-session-v1.jsonl @@ -0,0 +1,12 @@ +{"binding":{"episode":{"provenance":"unavailable","reasonCode":"caller-declared-no-session-evidence","ref":null},"run":{"provenance":"caller-asserted","ref":"run:sha256:7522f027d7bb67cbdbb3818345249dc1f4429fe8e484f44c6330083f1292e1d0"},"task":{"provenance":"caller-asserted","ref":"task:sha256:fce82f8f9ff62cd9af044e2e21bfd2a1822d4cc506bf174a285b8fd03fb011e9"},"workspace":{"provenance":"caller-asserted","ref":"workspace:sha256:50c688560321b967e32f410e861cf64cc7af45c817bddbe466aa1f703a29850e"}},"kind":"better-harness.experience-trace.header","producer":{"kind":"qoder-report-source-projection","platform":"qoder"},"recordOrdinal":0,"schemaVersion":1,"selection":{"analyzedCount":0,"confidence":"Low","eligibleCount":0,"representative":true,"sampled":false,"sourceFingerprint":"0123456789abcdef","strategy":"all-eligible","warningCodes":[]},"source":{"manifestSchemaVersion":2,"projectionDigest":"sha256:f46c5aaea639376da6fca7bfa9df215ee2a22bdad509761912a705d57d8eb9c9","reportSourceSchemaVersion":3},"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:task-episode","kind":"better-harness.experience-trace.event","payload":{"capability":"task-episode","reasonCode":"caller-declared-no-session-evidence"},"recordOrdinal":1,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:tool-observation","kind":"better-harness.experience-trace.event","payload":{"capability":"tool-observation","reasonCode":"caller-declared-no-session-evidence"},"recordOrdinal":2,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:validation-observation","kind":"better-harness.experience-trace.event","payload":{"capability":"validation-observation","reasonCode":"caller-declared-no-session-evidence"},"recordOrdinal":3,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:permission-boundary","kind":"better-harness.experience-trace.event","payload":{"capability":"permission-boundary","reasonCode":"caller-declared-no-session-evidence"},"recordOrdinal":4,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:human-approval","kind":"better-harness.experience-trace.event","payload":{"capability":"human-approval","reasonCode":"human-approval-not-observed"},"recordOrdinal":5,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:subagent-lineage","kind":"better-harness.experience-trace.event","payload":{"capability":"subagent-lineage","reasonCode":"subagent-parent-edge-not-observed"},"recordOrdinal":6,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:worktree-lineage","kind":"better-harness.experience-trace.event","payload":{"capability":"worktree-lineage","reasonCode":"worktree-lineage-not-observed"},"recordOrdinal":7,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:interruption-resume","kind":"better-harness.experience-trace.event","payload":{"capability":"interruption-resume","reasonCode":"interruption-resume-not-observed"},"recordOrdinal":8,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:runtime-stop","kind":"better-harness.experience-trace.event","payload":{"capability":"runtime-stop","reasonCode":"runtime-stop-not-observed"},"recordOrdinal":9,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"availability":"unavailable","eventType":"capability-gap","evidenceRef":"source:gap:component-snapshot","kind":"better-harness.experience-trace.event","payload":{"capability":"component-snapshot","reasonCode":"component-snapshot-not-bound"},"recordOrdinal":10,"schemaVersion":1,"traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} +{"coverage":[{"availability":"derived","capability":"task-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"derived","capability":"workspace-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"derived","capability":"run-binding","provenance":"caller-asserted","reasonCode":"explicit-caller-binding"},{"availability":"unavailable","capability":"task-episode","provenance":"unavailable","reasonCode":"caller-declared-no-session-evidence"},{"availability":"unavailable","capability":"tool-observation","provenance":"unavailable","reasonCode":"caller-declared-no-session-evidence"},{"availability":"unavailable","capability":"validation-observation","provenance":"unavailable","reasonCode":"caller-declared-no-session-evidence"},{"availability":"unavailable","capability":"permission-boundary","provenance":"unavailable","reasonCode":"caller-declared-no-session-evidence"},{"availability":"unavailable","capability":"human-approval","provenance":"unavailable","reasonCode":"human-approval-not-observed"},{"availability":"unavailable","capability":"subagent-lineage","provenance":"unavailable","reasonCode":"subagent-parent-edge-not-observed"},{"availability":"unavailable","capability":"worktree-lineage","provenance":"unavailable","reasonCode":"worktree-lineage-not-observed"},{"availability":"unavailable","capability":"interruption-resume","provenance":"unavailable","reasonCode":"interruption-resume-not-observed"},{"availability":"unavailable","capability":"runtime-stop","provenance":"unavailable","reasonCode":"runtime-stop-not-observed"},{"availability":"unavailable","capability":"component-snapshot","provenance":"unavailable","reasonCode":"component-snapshot-not-bound"}],"eventCount":10,"evidenceStatus":"unavailable","kind":"better-harness.experience-trace.terminal","policyUse":"advisory-only","recordCount":12,"recordOrdinal":11,"schemaVersion":1,"streamStatus":"complete","traceDigest":"sha256:e568040d4ce087a63abb593e760ea4b94b140a8da6421dad7b69ab583426956e","traceId":"trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"} diff --git a/scripts/better-harness-cli/cli.mjs b/scripts/better-harness-cli/cli.mjs index 6ec0153..01872f4 100644 --- a/scripts/better-harness-cli/cli.mjs +++ b/scripts/better-harness-cli/cli.mjs @@ -263,7 +263,7 @@ function hasHelpFlag(argv) { } function requiresOwnerHelpValidation(command, subcommand) { - return command === "harness" && subcommand === "source-review"; + return command === "harness" && (subcommand === "source-review" || subcommand === "experience-trace"); } function hasJsonFlag(argv) { diff --git a/scripts/better-harness-cli/registry.mjs b/scripts/better-harness-cli/registry.mjs index a8a14a3..29922d5 100644 --- a/scripts/better-harness-cli/registry.mjs +++ b/scripts/better-harness-cli/registry.mjs @@ -1,4 +1,5 @@ import { formatHostList, HOST_CAPABILITIES, hostIdsFor } from "../host-support/index.mjs"; +import { EXPERIENCE_TRACE_COMMAND_MANIFEST } from "../experience-trace/command-manifest.mjs"; import { PLUGIN_COMMAND_MANIFEST } from "../plugin-lifecycle/command-manifest.mjs"; export const FORMAT_VERSION = "1.0"; @@ -234,6 +235,13 @@ const COMMANDS = [ summary: "Create, compile, and apply a bounded report-source review.", description: "Expose an explicit local create, caller-authored decision, and confirmed apply lifecycle without calling a model or merging native evidence aliases into the outer evidence namespace.", }, + { + name: EXPERIENCE_TRACE_COMMAND_MANIFEST.name, + audience: EXPERIENCE_TRACE_COMMAND_MANIFEST.audience, + script: EXPERIENCE_TRACE_COMMAND_MANIFEST.entryScript, + summary: EXPERIENCE_TRACE_COMMAND_MANIFEST.summary, + description: EXPERIENCE_TRACE_COMMAND_MANIFEST.description, + }, { name: "task-loop-report", audience: "maintainer", diff --git a/scripts/experience-trace/cli.mjs b/scripts/experience-trace/cli.mjs new file mode 100644 index 0000000..a5eb213 --- /dev/null +++ b/scripts/experience-trace/cli.mjs @@ -0,0 +1,281 @@ +#!/usr/bin/env node + +import { open } from "node:fs/promises"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +import { EXPERIENCE_TRACE_COMMAND_MANIFEST } from "./command-manifest.mjs"; + +const SOURCE_MAX_BYTES = 16 * 1024 * 1024; +const TRACE_MAX_BYTES = 1024 * 1024; +const BINDING_KEY_RE = /^[A-Za-z0-9][A-Za-z0-9._-]{15,95}$/u; +const HELP_TOKENS = new Set(["--help", "-h"]); +const VALUE_OPTIONS = Object.freeze({ + create: new Set(["--source", "--task-key", "--workspace-key", "--run-key", "--episode-ref"]), + validate: new Set(["--trace"]), +}); +const FLAG_OPTIONS = Object.freeze({ + create: new Set(["--no-session-evidence", "--jsonl"]), + validate: new Set(), +}); + +const ERROR_DETAILS = Object.freeze({ + INVALID_USAGE: { exitCode: 64, message: "invalid experience-trace arguments" }, + MISSING_EPISODE_SELECTION: { exitCode: 64, message: "select exactly one episode mode" }, + INVALID_TRACE_BINDING: { exitCode: 1, message: "trace binding key is invalid" }, + SOURCE_READ_FAILED: { exitCode: 1, message: "unable to read report source" }, + TRACE_READ_FAILED: { exitCode: 1, message: "unable to read experience trace" }, + TRACE_BOUNDS_EXCEEDED: { exitCode: 1, message: "experience trace bounds exceeded" }, + INVALID_REPORT_SOURCE: { exitCode: 1, message: "report source is invalid" }, + UNSUPPORTED_TRACE_SOURCE_VERSION: { exitCode: 1, message: "report source version is unsupported" }, + UNSUPPORTED_TRACE_PLATFORM: { exitCode: 1, message: "report source platform is unsupported" }, + UNKNOWN_EPISODE_REF: { exitCode: 1, message: "selected episode is unavailable" }, + INVALID_EXPERIENCE_TRACE: { exitCode: 1, message: "experience trace is invalid" }, +}); + +const CREATE_RUNTIME_CODES = new Set([ + "TRACE_BOUNDS_EXCEEDED", + "INVALID_REPORT_SOURCE", + "UNSUPPORTED_TRACE_SOURCE_VERSION", + "UNSUPPORTED_TRACE_PLATFORM", + "UNKNOWN_EPISODE_REF", +]); + +class ExperienceTraceCliError extends Error { + constructor(code) { + super(code); + this.code = code; + } +} + +function fail(code) { + throw new ExperienceTraceCliError(code); +} + +function helpText() { + const { usage } = EXPERIENCE_TRACE_COMMAND_MANIFEST; + return [ + "Better Harness Experience Trace v1", + "", + "Create a bounded task-scoped trace from one explicit Qoder report source,", + "or validate one captured JSONL stream. Both phases are read-only.", + "", + "Usage:", + ` ${usage.create}`, + ` ${usage.validate}`, + "", + "Output:", + " create Canonical JSONL only", + " validate One canonical JSON validation document", + "", + "Options:", + " --source Explicit report-source input for create", + " --task-key Caller-owned task correlation key", + " --workspace-key Caller-owned workspace correlation key", + " --run-key Caller-owned run correlation key", + " --episode-ref Select one retained Qoder Episode", + " --no-session-evidence Declare that no native Episode applies", + " --jsonl Require JSONL output for create", + " --trace Explicit trace input for validate", + " -h, --help Print this help with no file reads", + "", + ].join("\n"); +} + +function isExactHelp(argv) { + return (argv.length === 1 && HELP_TOKENS.has(argv[0])) + || (argv.length === 2 && ["create", "validate"].includes(argv[0]) && HELP_TOKENS.has(argv[1])); +} + +function parseOptions(phase, argv) { + const options = Object.create(null); + const valueOptions = VALUE_OPTIONS[phase]; + const flagOptions = FLAG_OPTIONS[phase]; + + for (let index = 0; index < argv.length; index += 1) { + const token = argv[index]; + if (HELP_TOKENS.has(token) || !token.startsWith("--")) fail("INVALID_USAGE"); + if (flagOptions.has(token)) { + if (Object.hasOwn(options, token)) fail("INVALID_USAGE"); + options[token] = true; + continue; + } + if (!valueOptions.has(token)) fail("INVALID_USAGE"); + const value = argv[index + 1]; + if (value === undefined || value === "" || value.startsWith("--") || Object.hasOwn(options, token)) { + fail("INVALID_USAGE"); + } + options[token] = value; + index += 1; + } + + return options; +} + +export function parseExperienceTraceArgs(argv = []) { + if (isExactHelp(argv)) return { kind: "help" }; + + const [phase, ...rest] = argv; + if (phase !== "create" && phase !== "validate") fail("INVALID_USAGE"); + const options = parseOptions(phase, rest); + + if (phase === "create") { + for (const option of ["--source", "--task-key", "--workspace-key", "--run-key"]) { + if (!Object.hasOwn(options, option)) fail("INVALID_USAGE"); + } + if (!options["--jsonl"]) fail("INVALID_USAGE"); + if (options["--episode-ref"] && options["--no-session-evidence"]) fail("INVALID_USAGE"); + if (!options["--episode-ref"] && !options["--no-session-evidence"]) { + fail("MISSING_EPISODE_SELECTION"); + } + return { + kind: "create", + sourcePath: options["--source"], + taskKey: options["--task-key"], + workspaceKey: options["--workspace-key"], + runKey: options["--run-key"], + ...(options["--episode-ref"] ? { episodeRef: options["--episode-ref"] } : {}), + noSessionEvidence: Boolean(options["--no-session-evidence"]), + }; + } + + if (!Object.hasOwn(options, "--trace")) fail("INVALID_USAGE"); + return { kind: "validate", tracePath: options["--trace"] }; +} + +function assertBindingSyntax(options) { + if (![options.taskKey, options.workspaceKey, options.runKey].every((value) => BINDING_KEY_RE.test(value))) { + fail("INVALID_TRACE_BINDING"); + } +} + +async function readExplicitFile(filePath, maxBytes) { + let handle; + try { + handle = await open(filePath, "r"); + const bytes = Buffer.allocUnsafe(maxBytes + 1); + const { bytesRead } = await handle.read(bytes, 0, bytes.length, 0); + return bytes.subarray(0, bytesRead); + } finally { + if (handle) await handle.close(); + } +} + +function decodeUtf8(bytes) { + return new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes); +} + +function sourceRuntimeError(error) { + if (CREATE_RUNTIME_CODES.has(error?.code)) { + return new ExperienceTraceCliError(error.code); + } + return new ExperienceTraceCliError("INVALID_REPORT_SOURCE"); +} + +function traceRuntimeError(error) { + if (error?.code === "TRACE_BOUNDS_EXCEEDED") { + return new ExperienceTraceCliError(error.code); + } + return new ExperienceTraceCliError("INVALID_EXPERIENCE_TRACE"); +} + +async function loadRuntime() { + const [projectSource, contract] = await Promise.all([ + import("./project-source.mjs"), + import("./contract.mjs"), + ]); + return { ...projectSource, ...contract }; +} + +async function createTrace(options) { + assertBindingSyntax(options); + let sourceBytes; + try { + sourceBytes = await readExplicitFile(options.sourcePath, SOURCE_MAX_BYTES); + } catch { + fail("SOURCE_READ_FAILED"); + } + if (sourceBytes.length > SOURCE_MAX_BYTES) fail("TRACE_BOUNDS_EXCEEDED"); + + let source; + try { + source = JSON.parse(decodeUtf8(sourceBytes)); + } catch { + fail("INVALID_REPORT_SOURCE"); + } + + try { + const { createExperienceTrace, parseAndValidateExperienceTraceJsonl } = await loadRuntime(); + const created = await createExperienceTrace(source, { + taskKey: options.taskKey, + workspaceKey: options.workspaceKey, + runKey: options.runKey, + ...(options.episodeRef ? { episodeRef: options.episodeRef } : {}), + noSessionEvidence: options.noSessionEvidence, + }); + const jsonl = created?.jsonl; + if (typeof jsonl !== "string") fail("INVALID_REPORT_SOURCE"); + if (Buffer.byteLength(jsonl, "utf8") > TRACE_MAX_BYTES) fail("TRACE_BOUNDS_EXCEEDED"); + parseAndValidateExperienceTraceJsonl(Buffer.from(jsonl, "utf8")); + return jsonl; + } catch (error) { + if (error instanceof ExperienceTraceCliError) throw error; + throw sourceRuntimeError(error); + } +} + +async function validateTrace(options) { + let traceBytes; + try { + traceBytes = await readExplicitFile(options.tracePath, TRACE_MAX_BYTES); + } catch { + fail("TRACE_READ_FAILED"); + } + if (traceBytes.length > TRACE_MAX_BYTES) fail("TRACE_BOUNDS_EXCEEDED"); + + try { + const { + canonicalJson, + experienceTraceValidationDocument, + parseAndValidateExperienceTraceJsonl, + } = await loadRuntime(); + const records = parseAndValidateExperienceTraceJsonl(traceBytes); + const validation = experienceTraceValidationDocument(records); + return `${canonicalJson(validation)}\n`; + } catch (error) { + throw traceRuntimeError(error); + } +} + +function safeError(error) { + if (error instanceof ExperienceTraceCliError && ERROR_DETAILS[error.code]) return error; + return new ExperienceTraceCliError("INVALID_EXPERIENCE_TRACE"); +} + +export async function main(argv = process.argv.slice(2), { + stdout = process.stdout, + stderr = process.stderr, +} = {}) { + try { + const parsed = parseExperienceTraceArgs(argv); + if (parsed.kind === "help") { + stdout.write(helpText()); + return 0; + } + const output = parsed.kind === "create" + ? await createTrace(parsed) + : await validateTrace(parsed); + stdout.write(output); + return 0; + } catch (error) { + const safe = safeError(error); + const detail = ERROR_DETAILS[safe.code]; + stderr.write(`${safe.code}: ${detail.message}\n`); + return detail.exitCode; + } +} + +const currentFile = fileURLToPath(import.meta.url); +if (process.argv[1] && path.resolve(process.argv[1]) === currentFile) { + process.exitCode = await main(); +} diff --git a/scripts/experience-trace/command-manifest.mjs b/scripts/experience-trace/command-manifest.mjs new file mode 100644 index 0000000..65ecb9a --- /dev/null +++ b/scripts/experience-trace/command-manifest.mjs @@ -0,0 +1,50 @@ +const PROGRAM = "better-harness harness experience-trace"; + +function freezeRows(rows) { + return Object.freeze(rows.map((row) => Object.freeze({ ...row }))); +} + +export const EXPERIENCE_TRACE_COMMAND_MANIFEST = Object.freeze({ + capability: "experience-trace.v1", + name: "experience-trace", + entryScript: "experience-trace/cli.mjs", + audience: "maintainer", + effects: "read-only", + summary: "Create and validate task-scoped, privacy-safe Experience Trace JSONL.", + description: "Project a validated Qoder report source into a bounded, caller-bound Experience Trace stream, or validate a captured stream without reading any workspace or host state.", + usage: Object.freeze({ + create: `${PROGRAM} create --source --task-key --workspace-key --run-key (--episode-ref episode: | --no-session-evidence) --jsonl`, + validate: `${PROGRAM} validate --trace `, + }), + phases: freezeRows([ + { name: "create", stdout: "jsonl", reads: "explicit-source" }, + { name: "validate", stdout: "json", reads: "explicit-trace" }, + ]), + options: freezeRows([ + { name: "--source", phase: "create", value: "report.source.json" }, + { name: "--task-key", phase: "create", value: "opaque" }, + { name: "--workspace-key", phase: "create", value: "opaque" }, + { name: "--run-key", phase: "create", value: "opaque" }, + { name: "--episode-ref", phase: "create", value: "episode:" }, + { name: "--no-session-evidence", phase: "create", value: "none" }, + { name: "--jsonl", phase: "create", value: "none" }, + { name: "--trace", phase: "validate", value: "trace.jsonl" }, + ]), + examples: freezeRows([ + { phase: "create", argv: `${PROGRAM} create --source report.source.json --task-key --workspace-key --run-key --no-session-evidence --jsonl` }, + { phase: "validate", argv: `${PROGRAM} validate --trace trace.jsonl` }, + ]), + diagnostics: freezeRows([ + { code: "INVALID_USAGE", exitCode: 64 }, + { code: "MISSING_EPISODE_SELECTION", exitCode: 64 }, + { code: "INVALID_TRACE_BINDING", exitCode: 1 }, + { code: "SOURCE_READ_FAILED", exitCode: 1 }, + { code: "TRACE_READ_FAILED", exitCode: 1 }, + { code: "TRACE_BOUNDS_EXCEEDED", exitCode: 1 }, + { code: "INVALID_REPORT_SOURCE", exitCode: 1 }, + { code: "UNSUPPORTED_TRACE_SOURCE_VERSION", exitCode: 1 }, + { code: "UNSUPPORTED_TRACE_PLATFORM", exitCode: 1 }, + { code: "UNKNOWN_EPISODE_REF", exitCode: 1 }, + { code: "INVALID_EXPERIENCE_TRACE", exitCode: 1 }, + ]), +}); diff --git a/scripts/experience-trace/contract.mjs b/scripts/experience-trace/contract.mjs new file mode 100644 index 0000000..576eb86 --- /dev/null +++ b/scripts/experience-trace/contract.mjs @@ -0,0 +1,922 @@ +import { createHash } from "node:crypto"; + +export const EXPERIENCE_TRACE_SCHEMA_VERSION = 1; + +export const EXPERIENCE_TRACE_BOUNDS = Object.freeze({ + maxSourceBytes: 16 * 1024 * 1024, + maxTraceBytes: 1024 * 1024, + maxLineBytes: 64 * 1024, + maxRecords: 256, + maxEvents: 64, + maxDepth: 64, + maxValues: 100_000, +}); + +export const CAPABILITY_ORDER = Object.freeze([ + "task-binding", + "workspace-binding", + "run-binding", + "task-episode", + "tool-observation", + "validation-observation", + "permission-boundary", + "human-approval", + "subagent-lineage", + "worktree-lineage", + "interruption-resume", + "runtime-stop", + "component-snapshot", +]); + +export const FIXED_GAP_REASON_BY_CAPABILITY = Object.freeze({ + "human-approval": "human-approval-not-observed", + "subagent-lineage": "subagent-parent-edge-not-observed", + "worktree-lineage": "worktree-lineage-not-observed", + "interruption-resume": "interruption-resume-not-observed", + "runtime-stop": "runtime-stop-not-observed", + "component-snapshot": "component-snapshot-not-bound", +}); + +const HEADER_KIND = "better-harness.experience-trace.header"; +const EVENT_KIND = "better-harness.experience-trace.event"; +const TERMINAL_KIND = "better-harness.experience-trace.terminal"; +const VALIDATION_KIND = "better-harness.experience-trace.validation"; +const BINDING_PREFIX = "better-harness:experience-trace:binding:v1\0"; +const PROJECTION_PREFIX = "better-harness:experience-trace:source-projection:v1\0"; +const TRACE_ID_PREFIX = "better-harness:experience-trace:identity:v1\0"; +const TRACE_DIGEST_PREFIX = "better-harness:experience-trace:stream:v1\0"; +const CALLER_KEY_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{15,95}$/; +const HEX_16_PATTERN = /^[a-f0-9]{16}$/; +const HEX_20_PATTERN = /^[a-f0-9]{20}$/; +const HEX_24_PATTERN = /^[a-f0-9]{24}$/; +const EPISODE_REF_PATTERN = /^episode:[a-f0-9]{12,64}$/; +const SHA256_PATTERN = /^[a-f0-9]{64}$/; +const TRACE_ID_PATTERN = /^trace:sha256:[a-f0-9]{64}$/; +const SOURCE_DIGEST_PATTERN = /^sha256:[a-f0-9]{64}$/; +const BINDING_KINDS = new Set(["task", "workspace", "run"]); +const WARNING_CODES = new Set([ + "missing-required-root", + "missing-optional-root", + "disabled-source-root", + "partial-secret-scan-coverage", + "invalid-prior-learning-capture-report", +]); +const STRATEGIES = new Set(["stratified", "all-eligible", "latest-n"]); +const CONFIDENCES = new Set(["Low", "Medium", "High"]); +const CONTINUATIONS = new Set(["explicit", "session-bounded"]); +const START_BOUNDARIES = new Set([ + "session-start", + "explicit-task-key", + "explicit-boundary", + "idle-gap", + "progress-handoff", + "first-retained-boundary", +]); +const VALIDATION_CATEGORIES = new Set([ + "npm test", + "pnpm test", + "yarn test", + "node --test", + "vitest", + "jest", + "pytest", + "go test", + "cargo test", + "agent-lint", + "typecheck", + "lint", + "git diff --check", +]); +const VALIDATION_STATUSES = new Set(["passed", "failed", "observed"]); +const CLOSURE_STATUSES = new Set([ + "not-applicable", + "unobserved", + "closed", + "observed-without-pass", +]); +const REPAIR_STATUSES = new Set([ + "not-applicable", + "unobserved", + "review-required", + "repaired-and-passed", +]); +const GAP_REASON_CODES = new Set([ + "explicit-caller-binding", + "explicit-episode-selection", + "source-aggregate", + "source-observation", + "source-permission-summary", + "caller-declared-no-session-evidence", + "not-retained-by-source", + ...Object.values(FIXED_GAP_REASON_BY_CAPABILITY), +]); + +export class ExperienceTraceError extends Error { + constructor(code = "INVALID_EXPERIENCE_TRACE", message = "experience trace is invalid") { + super(message); + this.name = "ExperienceTraceError"; + this.code = code; + } +} + +export function failExperienceTrace(code = "INVALID_EXPERIENCE_TRACE", message = "experience trace is invalid") { + throw new ExperienceTraceError(code, message); +} + +function invalid() { + return failExperienceTrace("INVALID_EXPERIENCE_TRACE", "experience trace is invalid"); +} + +function invalidBinding() { + return failExperienceTrace("INVALID_TRACE_BINDING", "trace binding key is invalid"); +} + +function traceBoundsExceeded() { + return failExperienceTrace("TRACE_BOUNDS_EXCEEDED", "experience trace bounds exceeded"); +} + +export function compareCodeUnits(left, right) { + return left < right ? -1 : left > right ? 1 : 0; +} + +function isPlainObject(value) { + if (value === null || typeof value !== "object" || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +function assertNoLoneSurrogate(value) { + for (let index = 0; index < value.length; index += 1) { + const code = value.charCodeAt(index); + if (code >= 0xd800 && code <= 0xdbff) { + const next = value.charCodeAt(index + 1); + if (!(next >= 0xdc00 && next <= 0xdfff)) invalid(); + index += 1; + } else if (code >= 0xdc00 && code <= 0xdfff) { + invalid(); + } + } +} + +function assertJsonPrimitive(value) { + if (value === null || typeof value === "boolean") return; + if (typeof value === "string") { + assertNoLoneSurrogate(value); + return; + } + if (typeof value === "number" && Number.isSafeInteger(value) && !Object.is(value, -0)) return; + invalid(); +} + +/** + * Performs the depth and value-count preflight before any recursive serializer + * walks the supplied JSON value. The returned number is the number of JSON + * values, excluding object property names. + */ +export function assertIterativeJsonStructure(value, options = {}) { + const maxDepth = options.maxDepth ?? EXPERIENCE_TRACE_BOUNDS.maxDepth; + const maxValues = options.maxValues ?? EXPERIENCE_TRACE_BOUNDS.maxValues; + if (!Number.isSafeInteger(maxDepth) || maxDepth < 0 || !Number.isSafeInteger(maxValues) || maxValues < 1) { + invalid(); + } + + const stack = [{ value, depth: 0 }]; + const seen = new WeakSet(); + let count = 0; + while (stack.length > 0) { + const current = stack.pop(); + count += 1; + if (count > maxValues || current.depth > maxDepth) invalid(); + const currentValue = current.value; + if (currentValue === null || typeof currentValue !== "object") { + assertJsonPrimitive(currentValue); + continue; + } + if (!Array.isArray(currentValue) && !isPlainObject(currentValue)) invalid(); + if (seen.has(currentValue)) invalid(); + seen.add(currentValue); + if (Array.isArray(currentValue)) { + for (let index = currentValue.length - 1; index >= 0; index -= 1) { + stack.push({ value: currentValue[index], depth: current.depth + 1 }); + } + continue; + } + if (Object.getOwnPropertySymbols(currentValue).length > 0) invalid(); + const keys = Object.keys(currentValue); + for (const key of keys) assertNoLoneSurrogate(key); + for (let index = keys.length - 1; index >= 0; index -= 1) { + stack.push({ value: currentValue[keys[index]], depth: current.depth + 1 }); + } + } + return count; +} + +function canonicalString(value) { + assertNoLoneSurrogate(value); + return value.normalize("NFC"); +} + +function canonicalJsonValue(value) { + if (value === null) return "null"; + if (typeof value === "boolean") return value ? "true" : "false"; + if (typeof value === "number") return String(value); + if (typeof value === "string") return JSON.stringify(canonicalString(value)); + if (Array.isArray(value)) return `[${value.map(canonicalJsonValue).join(",")}]`; + + const entries = []; + const normalizedKeys = new Set(); + for (const key of Object.keys(value)) { + const normalizedKey = canonicalString(key); + if (normalizedKeys.has(normalizedKey)) invalid(); + normalizedKeys.add(normalizedKey); + entries.push([normalizedKey, value[key]]); + } + entries.sort((left, right) => compareCodeUnits(left[0], right[0])); + return `{${entries.map(([key, item]) => `${JSON.stringify(key)}:${canonicalJsonValue(item)}`).join(",")}}`; +} + +export function canonicalJson(value) { + assertIterativeJsonStructure(value); + return canonicalJsonValue(value); +} + +export function sha256Hex(value) { + if (typeof value !== "string" && !Buffer.isBuffer(value) && !(value instanceof Uint8Array)) invalid(); + return createHash("sha256").update(value).digest("hex"); +} + +function assertExactKeys(value, keys) { + if (!isPlainObject(value)) invalid(); + const actual = Object.keys(value).sort(compareCodeUnits); + const expected = [...keys].sort(compareCodeUnits); + if (actual.length !== expected.length || actual.some((key, index) => key !== expected[index])) invalid(); +} + +function assertString(value, pattern) { + if (typeof value !== "string") invalid(); + assertNoLoneSurrogate(value); + if (pattern && !pattern.test(value)) invalid(); +} + +function assertEnum(value, allowed) { + assertString(value); + if (!allowed.has(value)) invalid(); +} + +function assertSafeCount(value, minimum = 0, maximum = 1_000_000) { + if (!Number.isSafeInteger(value) || value < minimum || value > maximum) invalid(); +} + +function assertSortedUniqueStrings(values, allowed, maxLength) { + if (!Array.isArray(values) || values.length > maxLength) invalid(); + let previous = null; + for (const value of values) { + assertString(value); + if (allowed && !allowed.has(value)) invalid(); + if (previous !== null && compareCodeUnits(previous, value) >= 0) invalid(); + previous = value; + } +} + +function assertSortedUniqueTargetKeys(values) { + if (!Array.isArray(values) || values.length > 12) invalid(); + let previous = null; + for (const value of values) { + assertString(value, HEX_20_PATTERN); + if (previous !== null && compareCodeUnits(previous, value) >= 0) invalid(); + previous = value; + } +} + +function assertSelection(selection) { + assertExactKeys(selection, [ + "sourceFingerprint", "strategy", "eligibleCount", "analyzedCount", "sampled", + "representative", "confidence", "warningCodes", + ]); + assertString(selection.sourceFingerprint, HEX_16_PATTERN); + assertEnum(selection.strategy, STRATEGIES); + assertSafeCount(selection.eligibleCount); + assertSafeCount(selection.analyzedCount); + if (selection.analyzedCount > selection.eligibleCount || typeof selection.sampled !== "boolean" || + typeof selection.representative !== "boolean") invalid(); + if (selection.sampled !== (selection.analyzedCount < selection.eligibleCount)) invalid(); + if (selection.representative !== (!selection.sampled && selection.strategy !== "latest-n")) invalid(); + const expectedConfidence = selection.eligibleCount === 0 || selection.analyzedCount === 0 || selection.strategy === "latest-n" + ? "Low" + : selection.sampled ? "Medium" : "High"; + if (selection.confidence !== expectedConfidence || !CONFIDENCES.has(selection.confidence)) invalid(); + assertSortedUniqueStrings(selection.warningCodes, WARNING_CODES, 32); +} + +function assertChangeSets(changeSets) { + if (!Array.isArray(changeSets) || changeSets.length > 16) invalid(); + for (const changeSet of changeSets) { + assertExactKeys(changeSet, ["eventCount", "firstOrdinal", "lastOrdinal", "targetKeys"]); + assertSafeCount(changeSet.eventCount); + assertSafeCount(changeSet.firstOrdinal); + assertSafeCount(changeSet.lastOrdinal); + if (changeSet.firstOrdinal > changeSet.lastOrdinal) invalid(); + assertSortedUniqueTargetKeys(changeSet.targetKeys); + } +} + +function assertValidationSets(validationSets) { + if (!Array.isArray(validationSets) || validationSets.length > 32) invalid(); + let previousOrdinal = -1; + for (const validation of validationSets) { + assertExactKeys(validation, ["category", "status", "sourceOrdinal", "checkIdentity", "targetKeys"]); + assertEnum(validation.category, VALIDATION_CATEGORIES); + assertEnum(validation.status, VALIDATION_STATUSES); + assertSafeCount(validation.sourceOrdinal); + if (validation.sourceOrdinal <= previousOrdinal) invalid(); + previousOrdinal = validation.sourceOrdinal; + assertString(validation.checkIdentity, /^check:[a-f0-9]{24}$/); + assertSortedUniqueTargetKeys(validation.targetKeys); + } +} + +function assertPermissionBoundary(boundary) { + if (boundary === null) return; + assertExactKeys(boundary, ["prompted", "denied", "escalated", "protectedActions"]); + assertSafeCount(boundary.prompted); + assertSafeCount(boundary.denied); + assertSafeCount(boundary.escalated); + assertSafeCount(boundary.protectedActions); + if (boundary.protectedActions === 0 || boundary.prompted > boundary.protectedActions || + boundary.denied > boundary.protectedActions || boundary.escalated > boundary.protectedActions) invalid(); +} + +function assertProjection(projection) { + assertExactKeys(projection, ["schemaVersion", "producer", "selection", "episode", "absenceReason"]); + if (projection.schemaVersion !== EXPERIENCE_TRACE_SCHEMA_VERSION) invalid(); + assertExactKeys(projection.producer, ["platform", "reportSourceSchemaVersion", "manifestSchemaVersion"]); + if (projection.producer.platform !== "qoder" || projection.producer.reportSourceSchemaVersion !== 3 || + projection.producer.manifestSchemaVersion !== 2) invalid(); + assertSelection(projection.selection); + if (projection.episode === null) { + if (projection.absenceReason !== "caller-declared-no-session-evidence") invalid(); + return; + } + if (projection.absenceReason !== null) invalid(); + assertExactKeys(projection.episode, [ + "episodeRef", "sessionCount", "continuation", "startBoundary", "toolCallCount", "changeSets", + "validationSets", "permissionBoundary", "closureStatus", "repairStatus", + ]); + assertString(projection.episode.episodeRef, EPISODE_REF_PATTERN); + assertSafeCount(projection.episode.sessionCount, 1); + assertEnum(projection.episode.continuation, CONTINUATIONS); + if (projection.episode.sessionCount > 1 && projection.episode.continuation !== "explicit") invalid(); + assertEnum(projection.episode.startBoundary, START_BOUNDARIES); + assertSafeCount(projection.episode.toolCallCount); + assertChangeSets(projection.episode.changeSets); + assertValidationSets(projection.episode.validationSets); + assertPermissionBoundary(projection.episode.permissionBoundary); + assertEnum(projection.episode.closureStatus, CLOSURE_STATUSES); + assertEnum(projection.episode.repairStatus, REPAIR_STATUSES); +} + +export function bindingRefFromKey(kind, key) { + if (!BINDING_KINDS.has(kind) || typeof key !== "string" || !CALLER_KEY_PATTERN.test(key)) invalidBinding(); + return `${kind}:sha256:${sha256Hex(Buffer.from(`${BINDING_PREFIX}${kind}\0${key}`, "utf8"))}`; +} + +export function sourceProjectionDigestFor(projection) { + assertProjection(projection); + return `sha256:${sha256Hex(Buffer.from(`${PROJECTION_PREFIX}${canonicalJson(projection)}`, "utf8"))}`; +} + +function assertBindingRef(kind, value) { + assertString(value, new RegExp(`^${kind}:sha256:[a-f0-9]{64}$`)); +} + +export function traceIdFor(input) { + assertExactKeys(input, ["taskRef", "workspaceRef", "runRef", "sourceProjectionDigest", "episodeRef", "absenceReason"]); + assertBindingRef("task", input.taskRef); + assertBindingRef("workspace", input.workspaceRef); + assertBindingRef("run", input.runRef); + assertString(input.sourceProjectionDigest, SOURCE_DIGEST_PATTERN); + if (input.episodeRef === null) { + if (input.absenceReason !== "caller-declared-no-session-evidence") invalid(); + } else { + assertString(input.episodeRef, EPISODE_REF_PATTERN); + if (input.absenceReason !== null) invalid(); + } + const preimage = { + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + producerPlatform: "qoder", + taskRef: input.taskRef, + workspaceRef: input.workspaceRef, + runRef: input.runRef, + sourceProjectionDigest: input.sourceProjectionDigest, + episodeRef: input.episodeRef, + absenceReason: input.absenceReason, + }; + return `trace:sha256:${sha256Hex(Buffer.from(`${TRACE_ID_PREFIX}${canonicalJson(preimage)}`, "utf8"))}`; +} + +export function traceDigestFor(recordsWithoutTerminal) { + if (!Array.isArray(recordsWithoutTerminal) || recordsWithoutTerminal.length < 1) invalid(); + const lines = recordsWithoutTerminal.map((record) => { + if (!isPlainObject(record) || (record.kind !== HEADER_KIND && record.kind !== EVENT_KIND)) invalid(); + return canonicalJson(record); + }); + return `sha256:${sha256Hex(Buffer.from(`${TRACE_DIGEST_PREFIX}${lines.map((line) => `${line}\n`).join("")}`, "utf8"))}`; +} + +function equalJson(left, right) { + return canonicalJson(left) === canonicalJson(right); +} + +function sourceProjectionFromHeaderAndEvents(header, events) { + const noSession = header.binding.episode.ref === null; + if (noSession) { + return { + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + producer: { platform: "qoder", reportSourceSchemaVersion: 3, manifestSchemaVersion: 2 }, + selection: header.selection, + episode: null, + absenceReason: "caller-declared-no-session-evidence", + }; + } + if (events.length < 3 || events[0].eventType !== "task-episode" || events[1].eventType !== "tool-observation") invalid(); + const task = events[0]; + const tool = events[1]; + const validationSets = []; + let index = 2; + while (index < events.length && events[index].eventType === "validation-observation") { + const event = events[index]; + validationSets.push({ + category: event.payload.category, + status: event.payload.status, + sourceOrdinal: event.payload.sourceOrdinal, + checkIdentity: event.payload.checkIdentity, + targetKeys: event.payload.targetKeys, + }); + index += 1; + } + let permissionBoundary = null; + if (index < events.length && events[index].eventType === "permission-boundary") { + permissionBoundary = events[index].payload; + index += 1; + } + if (index >= events.length || events[index].eventType !== "episode-observation-ended") invalid(); + const end = events[index]; + return { + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + producer: { platform: "qoder", reportSourceSchemaVersion: 3, manifestSchemaVersion: 2 }, + selection: header.selection, + episode: { + episodeRef: task.payload.episodeRef, + sessionCount: task.payload.sessionCount, + continuation: task.payload.continuation, + startBoundary: task.payload.startBoundary, + toolCallCount: tool.payload.toolCallCount, + changeSets: tool.payload.changeSets.map(({ changeRef, ...changeSet }) => changeSet), + validationSets, + permissionBoundary, + closureStatus: end.payload.closureStatus, + repairStatus: end.payload.repairStatus, + }, + absenceReason: null, + }; +} + +function coverageFor(projection) { + const coverage = [ + { capability: "task-binding", availability: "derived", provenance: "caller-asserted", reasonCode: "explicit-caller-binding" }, + { capability: "workspace-binding", availability: "derived", provenance: "caller-asserted", reasonCode: "explicit-caller-binding" }, + { capability: "run-binding", availability: "derived", provenance: "caller-asserted", reasonCode: "explicit-caller-binding" }, + ]; + if (projection.episode === null) { + for (const capability of ["task-episode", "tool-observation", "validation-observation", "permission-boundary"]) { + coverage.push({ capability, availability: "unavailable", provenance: "unavailable", reasonCode: "caller-declared-no-session-evidence" }); + } + } else { + coverage.push( + { capability: "task-episode", availability: "observed", provenance: "source-projected", reasonCode: "explicit-episode-selection" }, + { capability: "tool-observation", availability: "derived", provenance: "source-projected", reasonCode: "source-aggregate" }, + projection.episode.validationSets.length > 0 + ? { capability: "validation-observation", availability: "observed", provenance: "source-projected", reasonCode: "source-observation" } + : { capability: "validation-observation", availability: "unavailable", provenance: "unavailable", reasonCode: "not-retained-by-source" }, + projection.episode.permissionBoundary !== null + ? { capability: "permission-boundary", availability: "derived", provenance: "source-projected", reasonCode: "source-permission-summary" } + : { capability: "permission-boundary", availability: "unavailable", provenance: "unavailable", reasonCode: "not-retained-by-source" }, + ); + } + for (const capability of Object.keys(FIXED_GAP_REASON_BY_CAPABILITY)) { + coverage.push({ capability, availability: "unavailable", provenance: "unavailable", reasonCode: FIXED_GAP_REASON_BY_CAPABILITY[capability] }); + } + if (!equalJson(coverage.map((row) => row.capability), CAPABILITY_ORDER)) invalid(); + return coverage; +} + +function expectedEvents(projection, traceId, coverage) { + const events = []; + const append = (eventType, availability, evidenceRef, payload) => { + events.push({ + kind: EVENT_KIND, + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + recordOrdinal: events.length + 1, + traceId, + eventType, + availability, + evidenceRef, + payload, + }); + }; + if (projection.episode !== null) { + const episode = projection.episode; + append("task-episode", "observed", "source:episode", { + episodeRef: episode.episodeRef, + sessionCount: episode.sessionCount, + continuation: episode.continuation, + startBoundary: episode.startBoundary, + }); + append("tool-observation", "derived", "source:tool-summary", { + toolCallCount: episode.toolCallCount, + changeSets: episode.changeSets.map((changeSet, index) => ({ changeRef: `source:change:${index + 1}`, ...changeSet })), + }); + for (let index = 0; index < episode.validationSets.length; index += 1) { + const validation = episode.validationSets[index]; + const validationRef = `source:validation:${index + 1}`; + append("validation-observation", "observed", validationRef, { validationRef, ...validation }); + } + if (episode.permissionBoundary !== null) { + append("permission-boundary", "derived", "source:permission-boundary", episode.permissionBoundary); + } + append("episode-observation-ended", "derived", "source:episode-end", { + closureStatus: episode.closureStatus, + repairStatus: episode.repairStatus, + }); + } + for (const row of coverage) { + if (row.availability === "unavailable") { + append("capability-gap", "unavailable", `source:gap:${row.capability}`, { + capability: row.capability, + reasonCode: row.reasonCode, + }); + } + } + return events; +} + +/** + * Builds the complete immutable wire stream from a source projection. Raw + * caller keys are consumed only for one-way binding references and never + * appear in a record or in the returned JSONL. + */ +export function createTraceFromProjection(input) { + try { + assertExactKeys(input, ["projection", "taskKey", "workspaceKey", "runKey"]); + const { projection, taskKey, workspaceKey, runKey } = input; + assertProjection(projection); + const taskRef = bindingRefFromKey("task", taskKey); + const workspaceRef = bindingRefFromKey("workspace", workspaceKey); + const runRef = bindingRefFromKey("run", runKey); + const sourceProjectionDigest = sourceProjectionDigestFor(projection); + const episodeRef = projection.episode === null ? null : projection.episode.episodeRef; + const traceId = traceIdFor({ + taskRef, + workspaceRef, + runRef, + sourceProjectionDigest, + episodeRef, + absenceReason: projection.absenceReason, + }); + const header = { + kind: HEADER_KIND, + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + recordOrdinal: 0, + traceId, + binding: { + task: { ref: taskRef, provenance: "caller-asserted" }, + workspace: { ref: workspaceRef, provenance: "caller-asserted" }, + run: { ref: runRef, provenance: "caller-asserted" }, + episode: projection.episode === null + ? { ref: null, provenance: "unavailable", reasonCode: "caller-declared-no-session-evidence" } + : { ref: episodeRef, provenance: "source-projected", reasonCode: "explicit-episode-selection" }, + }, + source: { + projectionDigest: sourceProjectionDigest, + reportSourceSchemaVersion: 3, + manifestSchemaVersion: 2, + }, + producer: { kind: "qoder-report-source-projection", platform: "qoder" }, + selection: projection.selection, + }; + const coverage = coverageFor(projection); + const events = expectedEvents(projection, traceId, coverage); + const withoutTerminal = [header, ...events]; + const terminal = { + kind: TERMINAL_KIND, + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + recordOrdinal: withoutTerminal.length, + traceId, + streamStatus: "complete", + evidenceStatus: projection.episode === null ? "unavailable" : "partial", + coverage, + eventCount: events.length, + recordCount: withoutTerminal.length + 1, + policyUse: "advisory-only", + traceDigest: traceDigestFor(withoutTerminal), + }; + const records = [...withoutTerminal, terminal]; + const jsonl = serializeExperienceTrace(records); + // Create must prove that its bounded canonical output is accepted by the + // exact byte-level validator used by the validate command. + parseAndValidateExperienceTraceJsonl(Buffer.from(jsonl, "utf8")); + return { records, jsonl }; + } catch (error) { + if (error instanceof ExperienceTraceError) throw error; + invalid(); + } +} + +function assertHeader(header) { + assertExactKeys(header, ["kind", "schemaVersion", "recordOrdinal", "traceId", "binding", "source", "producer", "selection"]); + if (header.kind !== HEADER_KIND || header.schemaVersion !== EXPERIENCE_TRACE_SCHEMA_VERSION || header.recordOrdinal !== 0) invalid(); + assertString(header.traceId, TRACE_ID_PATTERN); + assertExactKeys(header.binding, ["task", "workspace", "run", "episode"]); + for (const kind of ["task", "workspace", "run"]) { + assertExactKeys(header.binding[kind], ["ref", "provenance"]); + assertBindingRef(kind, header.binding[kind].ref); + if (header.binding[kind].provenance !== "caller-asserted") invalid(); + } + assertExactKeys(header.binding.episode, ["ref", "provenance", "reasonCode"]); + if (header.binding.episode.ref === null) { + if (header.binding.episode.provenance !== "unavailable" || header.binding.episode.reasonCode !== "caller-declared-no-session-evidence") invalid(); + } else { + assertString(header.binding.episode.ref, EPISODE_REF_PATTERN); + if (header.binding.episode.provenance !== "source-projected" || header.binding.episode.reasonCode !== "explicit-episode-selection") invalid(); + } + assertExactKeys(header.source, ["projectionDigest", "reportSourceSchemaVersion", "manifestSchemaVersion"]); + assertString(header.source.projectionDigest, SOURCE_DIGEST_PATTERN); + if (header.source.reportSourceSchemaVersion !== 3 || header.source.manifestSchemaVersion !== 2) invalid(); + assertExactKeys(header.producer, ["kind", "platform"]); + if (header.producer.kind !== "qoder-report-source-projection" || header.producer.platform !== "qoder") invalid(); + assertSelection(header.selection); +} + +function assertTerminalShape(terminal) { + assertExactKeys(terminal, [ + "kind", "schemaVersion", "recordOrdinal", "traceId", "streamStatus", "evidenceStatus", "coverage", + "eventCount", "recordCount", "policyUse", "traceDigest", + ]); + if (terminal.kind !== TERMINAL_KIND || terminal.schemaVersion !== EXPERIENCE_TRACE_SCHEMA_VERSION || + terminal.streamStatus !== "complete" || terminal.policyUse !== "advisory-only") invalid(); + assertString(terminal.traceId, TRACE_ID_PATTERN); + assertEnum(terminal.evidenceStatus, new Set(["partial", "unavailable"])); + assertString(terminal.traceDigest, SOURCE_DIGEST_PATTERN); +} + +function assertRecordShapes(records) { + if (!Array.isArray(records)) invalid(); + if (records.length > EXPERIENCE_TRACE_BOUNDS.maxRecords) traceBoundsExceeded(); + if (records.length < 3) invalid(); + if (records.length - 2 > EXPERIENCE_TRACE_BOUNDS.maxEvents) traceBoundsExceeded(); + let values = 0; + for (const record of records) { + values += assertIterativeJsonStructure(record, { maxDepth: EXPERIENCE_TRACE_BOUNDS.maxDepth, maxValues: EXPERIENCE_TRACE_BOUNDS.maxValues }); + if (values > EXPERIENCE_TRACE_BOUNDS.maxValues) invalid(); + } + if (records[0]?.kind !== HEADER_KIND || records.at(-1)?.kind !== TERMINAL_KIND) invalid(); + for (let index = 0; index < records.length; index += 1) { + if (!isPlainObject(records[index]) || records[index].recordOrdinal !== index) invalid(); + } +} + +function validateExperienceTraceRecordsInternal(records) { + assertRecordShapes(records); + const header = records[0]; + const terminal = records.at(-1); + const events = records.slice(1, -1); + assertHeader(header); + assertTerminalShape(terminal); + for (const event of events) { + assertExactKeys(event, ["kind", "schemaVersion", "recordOrdinal", "traceId", "eventType", "availability", "evidenceRef", "payload"]); + if (event.kind !== EVENT_KIND || event.schemaVersion !== EXPERIENCE_TRACE_SCHEMA_VERSION || event.traceId !== header.traceId) invalid(); + assertEnum(event.availability, new Set(["observed", "derived", "unavailable"])); + assertString(event.evidenceRef, /^source:(episode|tool-summary|validation:[1-9][0-9]*|permission-boundary|episode-end|gap:[a-z-]+)$/); + } + if (terminal.traceId !== header.traceId || terminal.recordOrdinal !== records.length - 1 || terminal.eventCount !== events.length || + terminal.recordCount !== records.length) invalid(); + const projection = sourceProjectionFromHeaderAndEvents(header, events); + assertProjection(projection); + const projectedEpisodeRef = projection.episode === null ? null : projection.episode.episodeRef; + if (header.binding.episode.ref !== projectedEpisodeRef) invalid(); + const coverage = coverageFor(projection); + const expected = expectedEvents(projection, header.traceId, coverage); + if (!equalJson(events, expected)) invalid(); + const sourceDigest = sourceProjectionDigestFor(projection); + if (header.source.projectionDigest !== sourceDigest) invalid(); + const expectedTraceId = traceIdFor({ + taskRef: header.binding.task.ref, + workspaceRef: header.binding.workspace.ref, + runRef: header.binding.run.ref, + sourceProjectionDigest: sourceDigest, + episodeRef: projection.episode === null ? null : projection.episode.episodeRef, + absenceReason: projection.absenceReason, + }); + if (header.traceId !== expectedTraceId) invalid(); + const expectedEvidenceStatus = projection.episode === null ? "unavailable" : "partial"; + if (terminal.evidenceStatus !== expectedEvidenceStatus || !equalJson(terminal.coverage, coverage)) invalid(); + const digest = traceDigestFor(records.slice(0, -1)); + if (terminal.traceDigest !== digest) invalid(); + return records; +} + +export function validateExperienceTraceRecords(records) { + try { + return validateExperienceTraceRecordsInternal(records); + } catch (error) { + if (error instanceof ExperienceTraceError) throw error; + invalid(); + } +} + +function assertNoDuplicateJsonObjectKeys(text) { + let index = 0; + const skipWhitespace = () => { + while (/[ \t\n\r]/u.test(text[index] ?? "")) index += 1; + }; + const parseString = () => { + const start = index; + index += 1; + while (index < text.length) { + const character = text[index]; + if (character === "\\") { + index += 1; + if (text[index] === "u") index += 4; + index += 1; + } else if (character === "\"") { + index += 1; + return JSON.parse(text.slice(start, index)); + } else { + index += 1; + } + } + invalid(); + }; + const parseValue = () => { + skipWhitespace(); + if (text[index] === "\"") { + parseString(); + return; + } + if (text[index] === "{") { + index += 1; + skipWhitespace(); + const keys = new Set(); + if (text[index] === "}") { + index += 1; + return; + } + while (true) { + skipWhitespace(); + if (text[index] !== "\"") invalid(); + const key = parseString(); + if (keys.has(key)) invalid(); + keys.add(key); + skipWhitespace(); + if (text[index] !== ":") invalid(); + index += 1; + parseValue(); + skipWhitespace(); + if (text[index] === "}") { + index += 1; + return; + } + if (text[index] !== ",") invalid(); + index += 1; + } + } + if (text[index] === "[") { + index += 1; + skipWhitespace(); + if (text[index] === "]") { + index += 1; + return; + } + while (true) { + parseValue(); + skipWhitespace(); + if (text[index] === "]") { + index += 1; + return; + } + if (text[index] !== ",") invalid(); + index += 1; + } + } + const number = text.slice(index).match(/^-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?/u); + if (number) { + index += number[0].length; + return; + } + for (const literal of ["true", "false", "null"]) { + if (text.startsWith(literal, index)) { + index += literal.length; + return; + } + } + invalid(); + }; + parseValue(); + skipWhitespace(); + if (index !== text.length) invalid(); +} + +function experienceTraceInputBytes(input) { + if (Buffer.isBuffer(input)) return input; + if (input instanceof Uint8Array) return Buffer.from(input.buffer, input.byteOffset, input.byteLength); + if (typeof input !== "string") invalid(); + // A JavaScript string has no raw UTF-8 representation to scan. Preserve the + // string contract by rejecting lone surrogates before Buffer would replace + // them with U+FFFD, then apply the same byte bounds to its exact UTF-8 bytes. + assertNoLoneSurrogate(input); + return Buffer.from(input, "utf8"); +} + +function preflightExperienceTraceBytes(bytes) { + if (bytes.length > EXPERIENCE_TRACE_BOUNDS.maxTraceBytes) traceBoundsExceeded(); + const lineRanges = []; + let lineStart = 0; + let lineBytes = 0; + for (let index = 0; index < bytes.length; index += 1) { + lineBytes += 1; + if (lineBytes > EXPERIENCE_TRACE_BOUNDS.maxLineBytes) traceBoundsExceeded(); + if (bytes[index] !== 0x0a) continue; + lineRanges.push([lineStart, index]); + if (lineRanges.length > EXPERIENCE_TRACE_BOUNDS.maxRecords || + lineRanges.length - 2 > EXPERIENCE_TRACE_BOUNDS.maxEvents) { + traceBoundsExceeded(); + } + lineStart = index + 1; + lineBytes = 0; + } + const hasFinalLf = lineStart === bytes.length; + const physicalLineCount = lineRanges.length + (hasFinalLf ? 0 : 1); + if (physicalLineCount > EXPERIENCE_TRACE_BOUNDS.maxRecords || + physicalLineCount - 2 > EXPERIENCE_TRACE_BOUNDS.maxEvents) { + traceBoundsExceeded(); + } + return { lineRanges, hasFinalLf }; +} + +function assertValidUtf8(bytes) { + const decoded = bytes.toString("utf8"); + if (!Buffer.from(decoded, "utf8").equals(bytes)) invalid(); +} + +export function parseAndValidateExperienceTraceJsonl(input) { + const bytes = experienceTraceInputBytes(input); + const preflight = preflightExperienceTraceBytes(bytes); + // Bounds deliberately win over malformed UTF-8 for raw byte inputs. Decode + // and round-trip only after the complete single-pass byte preflight. + assertValidUtf8(bytes); + if (bytes.length === 0 || bytes.subarray(0, 3).equals(Buffer.from([0xef, 0xbb, 0xbf])) || bytes.includes(0x0d) || + !preflight.hasFinalLf || preflight.lineRanges.length < 2) invalid(); + const records = []; + for (const [lineStart, lineEnd] of preflight.lineRanges) { + if (lineStart === lineEnd) invalid(); + const line = bytes.toString("utf8", lineStart, lineEnd); + let record; + try { + record = JSON.parse(line); + } catch { + invalid(); + } + assertIterativeJsonStructure(record); + assertNoDuplicateJsonObjectKeys(line); + if (!Buffer.from(`${canonicalJson(record)}\n`, "utf8").equals(bytes.subarray(lineStart, lineEnd + 1))) invalid(); + records.push(record); + } + validateExperienceTraceRecords(records); + return records; +} + +export function serializeExperienceTrace(records) { + validateExperienceTraceRecords(records); + const lines = records.map((record) => `${canonicalJson(record)}\n`); + const bytes = Buffer.from(lines.join(""), "utf8"); + if (bytes.length > EXPERIENCE_TRACE_BOUNDS.maxTraceBytes || lines.some((line) => Buffer.byteLength(line, "utf8") > EXPERIENCE_TRACE_BOUNDS.maxLineBytes)) { + traceBoundsExceeded(); + } + return bytes.toString("utf8"); +} + +export function experienceTraceValidationDocument(records) { + validateExperienceTraceRecords(records); + const terminal = records.at(-1); + return { + kind: VALIDATION_KIND, + schemaVersion: EXPERIENCE_TRACE_SCHEMA_VERSION, + valid: true, + traceId: terminal.traceId, + streamStatus: terminal.streamStatus, + evidenceStatus: terminal.evidenceStatus, + recordCount: terminal.recordCount, + eventCount: terminal.eventCount, + traceDigest: terminal.traceDigest, + }; +} diff --git a/scripts/experience-trace/index.mjs b/scripts/experience-trace/index.mjs new file mode 100644 index 0000000..6051f53 --- /dev/null +++ b/scripts/experience-trace/index.mjs @@ -0,0 +1,31 @@ +// Public behavioral import surface for the experience-trace capability. +// Cross-capability consumers must import from this file instead of reaching +// into the contract or Qoder report-source projection internals. + +export { + CAPABILITY_ORDER, + EXPERIENCE_TRACE_BOUNDS, + EXPERIENCE_TRACE_SCHEMA_VERSION, + ExperienceTraceError, + FIXED_GAP_REASON_BY_CAPABILITY, + assertIterativeJsonStructure, + bindingRefFromKey, + canonicalJson, + compareCodeUnits, + createTraceFromProjection, + experienceTraceValidationDocument, + failExperienceTrace, + parseAndValidateExperienceTraceJsonl, + serializeExperienceTrace, + sha256Hex, + sourceProjectionDigestFor, + traceDigestFor, + traceIdFor, + validateExperienceTraceRecords, +} from "./contract.mjs"; + +export { + createExperienceTrace, + ExperienceTraceSourceError, + projectQoderReportSource, +} from "./project-source.mjs"; diff --git a/scripts/experience-trace/project-source.mjs b/scripts/experience-trace/project-source.mjs new file mode 100644 index 0000000..5778cac --- /dev/null +++ b/scripts/experience-trace/project-source.mjs @@ -0,0 +1,407 @@ +import { validateHarnessReportSource } from "../harness-analysis/report-source/index.mjs"; +import { + bindingRefFromKey, + createTraceFromProjection, + EXPERIENCE_TRACE_BOUNDS, +} from "./contract.mjs"; + +const SOURCE_SCHEMA_VERSION = 3; +const MANIFEST_SCHEMA_VERSION = 2; +const MAX_SOURCE_DEPTH = EXPERIENCE_TRACE_BOUNDS.maxDepth; +const MAX_SOURCE_VALUES = EXPERIENCE_TRACE_BOUNDS.maxValues; +const MAX_SAFE_COUNT = 1_000_000; +const MAX_WARNING_CODES = 32; +const MAX_CHANGE_SETS = 16; +const MAX_VALIDATION_SETS = 32; +const MAX_TARGET_KEYS = 12; + +const WARNING_CODES = new Set([ + "missing-required-root", + "missing-optional-root", + "disabled-source-root", + "partial-secret-scan-coverage", + "invalid-prior-learning-capture-report", +]); +const STRATEGIES = new Set(["stratified", "all-eligible", "latest-n"]); +const CONTINUATIONS = new Set(["explicit", "session-bounded"]); +const START_BOUNDARIES = new Set([ + "session-start", + "explicit-task-key", + "explicit-boundary", + "idle-gap", + "progress-handoff", + "first-retained-boundary", +]); +const VALIDATION_CATEGORIES = new Set([ + "npm test", + "pnpm test", + "yarn test", + "node --test", + "vitest", + "jest", + "pytest", + "go test", + "cargo test", + "agent-lint", + "typecheck", + "lint", + "git diff --check", +]); +const VALIDATION_STATUSES = new Set(["passed", "failed", "observed"]); +const CLOSURE_STATUSES = new Set(["not-applicable", "unobserved", "closed", "observed-without-pass"]); +const REPAIR_STATUSES = new Set(["not-applicable", "unobserved", "review-required", "repaired-and-passed"]); + +const EPISODE_REF_RE = /^episode:[a-f0-9]{12,64}$/u; +const SOURCE_FINGERPRINT_RE = /^[a-f0-9]{16}$/u; +const TARGET_KEY_RE = /^[a-f0-9]{20}$/u; +const CHECK_IDENTITY_RE = /^check:[a-f0-9]{24}$/u; + +/** + * A deliberately value-free error boundary for callers that map contract codes + * onto stable CLI output. Nothing from an untrusted report source is retained + * in the error message. + */ +export class ExperienceTraceSourceError extends Error { + constructor(code) { + super(code); + this.name = "ExperienceTraceSourceError"; + this.code = code; + } +} + +function fail(code) { + throw new ExperienceTraceSourceError(code); +} + +function isRecord(value) { + return Boolean(value) && typeof value === "object" && !Array.isArray(value); +} + +function own(record, field) { + if (!isRecord(record) || !Object.hasOwn(record, field)) fail("INVALID_REPORT_SOURCE"); + return record[field]; +} + +function requiredRecord(record, field) { + const value = own(record, field); + if (!isRecord(value)) fail("INVALID_REPORT_SOURCE"); + return value; +} + +function requiredArray(record, field) { + const value = own(record, field); + if (!Array.isArray(value)) fail("INVALID_REPORT_SOURCE"); + return value; +} + +function requiredString(record, field) { + const value = own(record, field); + if (typeof value !== "string") fail("INVALID_REPORT_SOURCE"); + return value; +} + +function requiredBoolean(record, field) { + const value = own(record, field); + if (typeof value !== "boolean") fail("INVALID_REPORT_SOURCE"); + return value; +} + +function requiredCount(record, field, { minimum = 0 } = {}) { + const value = own(record, field); + if (!Number.isSafeInteger(value) || value < minimum || value > MAX_SAFE_COUNT) { + fail("INVALID_REPORT_SOURCE"); + } + return value; +} + +function requiredEnum(record, field, values) { + const value = requiredString(record, field); + if (!values.has(value)) fail("INVALID_REPORT_SOURCE"); + return value; +} + +function compareCodeUnits(left, right) { + if (left < right) return -1; + if (left > right) return 1; + return 0; +} + +function canonicalStringSet(value, { maxLength, matcher }) { + if (!Array.isArray(value) || value.length > maxLength) fail("INVALID_REPORT_SOURCE"); + const output = new Set(); + for (const item of value) { + if (typeof item !== "string" || !matcher(item)) fail("INVALID_REPORT_SOURCE"); + output.add(item); + } + return [...output].sort(compareCodeUnits); +} + +/** + * Walk an already parsed source without recursive validators. This models the + * JSON tree, not JavaScript object graphs: cycles, non-JSON leaves, and holes + * are rejected rather than risking unbounded work in a later product validator. + */ +export function preflightQoderReportSource(source) { + try { + const seen = new WeakSet(); + const stack = [{ value: source, depth: 0 }]; + let valueCount = 0; + + while (stack.length > 0) { + const { value, depth } = stack.pop(); + valueCount += 1; + if (valueCount > MAX_SOURCE_VALUES || depth > MAX_SOURCE_DEPTH) { + fail("INVALID_REPORT_SOURCE"); + } + + if (value === null || typeof value === "string" || typeof value === "boolean") continue; + if (typeof value === "number") { + if (!Number.isFinite(value)) fail("INVALID_REPORT_SOURCE"); + continue; + } + if (typeof value !== "object") fail("INVALID_REPORT_SOURCE"); + if (seen.has(value)) fail("INVALID_REPORT_SOURCE"); + seen.add(value); + + if (Array.isArray(value)) { + for (let index = value.length - 1; index >= 0; index -= 1) { + if (!Object.hasOwn(value, index)) fail("INVALID_REPORT_SOURCE"); + stack.push({ value: value[index], depth: depth + 1 }); + } + continue; + } + + if (Object.getOwnPropertySymbols(value).length > 0) fail("INVALID_REPORT_SOURCE"); + const keys = Object.keys(value); + for (let index = keys.length - 1; index >= 0; index -= 1) { + const key = keys[index]; + stack.push({ value: value[key], depth: depth + 1 }); + } + } + } catch (error) { + if (error instanceof ExperienceTraceSourceError) throw error; + fail("INVALID_REPORT_SOURCE"); + } +} + +function assertMarkerCompatibility(source) { + if (!isRecord(source) || !isRecord(source.manifest) || !isRecord(source.manifest.scope)) { + fail("INVALID_REPORT_SOURCE"); + } + + const manifest = source.manifest; + const scope = manifest.scope; + if (!Number.isInteger(source.schemaVersion) || !Number.isInteger(manifest.schemaVersion)) { + fail("INVALID_REPORT_SOURCE"); + } + if (typeof source.kind !== "string" || source.kind !== "harness-report-source" + || typeof manifest.kind !== "string" || manifest.kind !== "session-observation-manifest") { + fail("INVALID_REPORT_SOURCE"); + } + if (source.schemaVersion !== SOURCE_SCHEMA_VERSION || manifest.schemaVersion !== MANIFEST_SCHEMA_VERSION) { + fail("UNSUPPORTED_TRACE_SOURCE_VERSION"); + } + if (typeof scope.platform !== "string" || typeof scope.workspaceScope !== "string") { + fail("INVALID_REPORT_SOURCE"); + } + if (scope.platform !== "qoder" || scope.workspaceScope !== "workspace") { + fail("UNSUPPORTED_TRACE_PLATFORM"); + } +} + +function assertBroadSourceValidity(source) { + let errors; + try { + errors = validateHarnessReportSource(source); + } catch { + fail("INVALID_REPORT_SOURCE"); + } + if (!Array.isArray(errors) || errors.length > 0) fail("INVALID_REPORT_SOURCE"); +} + +function projectSelection(source) { + const manifest = requiredRecord(source, "manifest"); + const sources = requiredRecord(manifest, "sources"); + const selection = requiredRecord(manifest, "selection"); + const sourceFingerprint = requiredString(sources, "fingerprint"); + if (!SOURCE_FINGERPRINT_RE.test(sourceFingerprint)) fail("INVALID_REPORT_SOURCE"); + + const strategy = requiredEnum(selection, "strategy", STRATEGIES); + const eligibleCount = requiredCount(selection, "eligibleCount"); + const analyzedCount = requiredCount(selection, "analyzedCount"); + if (analyzedCount > eligibleCount) fail("INVALID_REPORT_SOURCE"); + const sampled = requiredBoolean(selection, "sampled"); + const representative = requiredBoolean(selection, "representative"); + const confidence = requiredString(selection, "confidence"); + const expectedConfidence = eligibleCount === 0 || analyzedCount === 0 || strategy === "latest-n" + ? "Low" + : analyzedCount < eligibleCount ? "Medium" : "High"; + if (sampled !== (analyzedCount < eligibleCount) + || representative !== (!sampled && strategy !== "latest-n") + || confidence !== expectedConfidence) { + fail("INVALID_REPORT_SOURCE"); + } + + const warningCodes = canonicalStringSet(own(manifest, "warningCodes"), { + maxLength: MAX_WARNING_CODES, + matcher: (value) => WARNING_CODES.has(value), + }); + return { + sourceFingerprint, + strategy, + eligibleCount, + analyzedCount, + sampled, + representative, + confidence, + warningCodes, + }; +} + +function projectTargetKeys(record) { + return canonicalStringSet(requiredArray(record, "targetKeys"), { + maxLength: MAX_TARGET_KEYS, + matcher: (value) => TARGET_KEY_RE.test(value), + }); +} + +function projectChangeSets(episode) { + const changeSets = requiredArray(episode, "changeSets"); + if (changeSets.length > MAX_CHANGE_SETS) fail("INVALID_REPORT_SOURCE"); + return changeSets.map((change) => { + if (!isRecord(change)) fail("INVALID_REPORT_SOURCE"); + const eventCount = requiredCount(change, "eventCount"); + const firstOrdinal = requiredCount(change, "firstOrdinal"); + const lastOrdinal = requiredCount(change, "lastOrdinal"); + if (firstOrdinal > lastOrdinal) fail("INVALID_REPORT_SOURCE"); + return { + eventCount, + firstOrdinal, + lastOrdinal, + targetKeys: projectTargetKeys(change), + }; + }); +} + +function projectValidationSets(episode) { + const validationSets = requiredArray(episode, "validationSets"); + if (validationSets.length > MAX_VALIDATION_SETS) fail("INVALID_REPORT_SOURCE"); + const seenOrdinals = new Set(); + const projected = validationSets.map((validation) => { + if (!isRecord(validation)) fail("INVALID_REPORT_SOURCE"); + const sourceOrdinal = requiredCount(validation, "ordinal"); + if (seenOrdinals.has(sourceOrdinal)) fail("INVALID_REPORT_SOURCE"); + seenOrdinals.add(sourceOrdinal); + const checkIdentity = requiredString(validation, "checkIdentity"); + if (!CHECK_IDENTITY_RE.test(checkIdentity)) fail("INVALID_REPORT_SOURCE"); + return { + category: requiredEnum(validation, "category", VALIDATION_CATEGORIES), + status: requiredEnum(validation, "status", VALIDATION_STATUSES), + sourceOrdinal, + checkIdentity, + targetKeys: projectTargetKeys(validation), + }; + }); + return projected.sort((left, right) => left.sourceOrdinal - right.sourceOrdinal); +} + +function projectPermissionBoundary(episode) { + if (!Object.hasOwn(episode, "permissionSummary")) return null; + const summary = requiredRecord(episode, "permissionSummary"); + const prompted = requiredCount(summary, "prompted"); + const denied = requiredCount(summary, "denied"); + const escalated = requiredCount(summary, "escalated"); + const protectedActions = requiredCount(summary, "protectedActions", { minimum: 1 }); + if (prompted > protectedActions || denied > protectedActions || escalated > protectedActions) { + fail("INVALID_REPORT_SOURCE"); + } + return { prompted, denied, escalated, protectedActions }; +} + +function projectEpisode(source, episodeRef) { + if (!EPISODE_REF_RE.test(episodeRef)) fail("UNKNOWN_EPISODE_REF"); + const episodes = requiredArray(source, "taskEpisodes"); + const matches = episodes.filter((episode) => isRecord(episode) && episode.id === episodeRef); + if (matches.length !== 1) fail("UNKNOWN_EPISODE_REF"); + const episode = matches[0]; + + const selectedEpisodeRef = requiredString(episode, "id"); + if (!EPISODE_REF_RE.test(selectedEpisodeRef)) fail("INVALID_REPORT_SOURCE"); + const sessionCount = requiredCount(episode, "sessionCount", { minimum: 1 }); + const continuation = requiredEnum(episode, "continuation", CONTINUATIONS); + if (sessionCount > 1 && continuation !== "explicit") fail("INVALID_REPORT_SOURCE"); + + return { + episodeRef: selectedEpisodeRef, + sessionCount, + continuation, + startBoundary: requiredEnum(episode, "startBoundary", START_BOUNDARIES), + toolCallCount: requiredCount(episode, "toolCalls"), + changeSets: projectChangeSets(episode), + validationSets: projectValidationSets(episode), + permissionBoundary: projectPermissionBoundary(episode), + closureStatus: requiredEnum(requiredRecord(episode, "closure"), "status", CLOSURE_STATUSES), + repairStatus: requiredEnum(requiredRecord(episode, "repair"), "status", REPAIR_STATUSES), + }; +} + +function normalizeSelectionOptions(options) { + if (!isRecord(options)) fail("INVALID_USAGE"); + const noSessionEvidence = options.noSessionEvidence === true; + if (options.noSessionEvidence !== undefined && typeof options.noSessionEvidence !== "boolean") { + fail("INVALID_USAGE"); + } + if (noSessionEvidence && options.episodeRef !== undefined) fail("INVALID_USAGE"); + if (noSessionEvidence) return { noSessionEvidence: true, episodeRef: null }; + if (!Object.hasOwn(options, "episodeRef") || typeof options.episodeRef !== "string") { + fail("MISSING_EPISODE_SELECTION"); + } + return { noSessionEvidence: false, episodeRef: options.episodeRef }; +} + +/** + * Project a validated Qoder report source into the trace's closed, reader-safe + * source shape. This is intentionally an allowlist construction; no source + * object, evidence locator, session identity, or unknown field is copied. + */ +export function projectQoderReportSource(source, options = {}) { + try { + preflightQoderReportSource(source); + assertMarkerCompatibility(source); + assertBroadSourceValidity(source); + // Source compatibility and integrity are established before selection is + // interpreted, so a malformed caller ref cannot mask a source marker. + const selectionOptions = normalizeSelectionOptions(options); + + return { + schemaVersion: 1, + producer: { + platform: "qoder", + reportSourceSchemaVersion: SOURCE_SCHEMA_VERSION, + manifestSchemaVersion: MANIFEST_SCHEMA_VERSION, + }, + selection: projectSelection(source), + episode: selectionOptions.noSessionEvidence ? null : projectEpisode(source, selectionOptions.episodeRef), + absenceReason: selectionOptions.noSessionEvidence ? "caller-declared-no-session-evidence" : null, + }; + } catch (error) { + if (error instanceof ExperienceTraceSourceError) throw error; + fail("INVALID_REPORT_SOURCE"); + } +} + +export function createExperienceTrace(source, { + taskKey, + workspaceKey, + runKey, + episodeRef, + noSessionEvidence, +} = {}) { + // Preserve the public error precedence: caller binding syntax fails before a + // source is traversed. The constructor re-derives these one-way references. + bindingRefFromKey("task", taskKey); + bindingRefFromKey("workspace", workspaceKey); + bindingRefFromKey("run", runKey); + const projection = projectQoderReportSource(source, { episodeRef, noSessionEvidence }); + return createTraceFromProjection({ projection, taskKey, workspaceKey, runKey }); +} diff --git a/scripts/npm-package/verify-pack.mjs b/scripts/npm-package/verify-pack.mjs index e6916e5..3d62f04 100644 --- a/scripts/npm-package/verify-pack.mjs +++ b/scripts/npm-package/verify-pack.mjs @@ -190,6 +190,11 @@ const required = [ "package/scripts/harness-analysis/learning-loop-review-packet.mjs", "package/scripts/harness-analysis/record-fix-output.mjs", "package/scripts/harness-analysis/preview-support/canvas-transform.mjs", + "package/scripts/experience-trace/cli.mjs", + "package/scripts/experience-trace/command-manifest.mjs", + "package/scripts/experience-trace/contract.mjs", + "package/scripts/experience-trace/index.mjs", + "package/scripts/experience-trace/project-source.mjs", "package/scripts/session-analysis/episode-facts.mjs", "package/scripts/session-analysis/result-facts.mjs", "package/scripts/session-analysis/session-core-facts.mjs", @@ -320,6 +325,11 @@ const requiredBundleEntries = [ "scripts/harness-analysis/learning-loop-review-packet.mjs", "scripts/harness-analysis/record-fix-output.mjs", "scripts/harness-analysis/preview-support/canvas-transform.mjs", + "scripts/experience-trace/cli.mjs", + "scripts/experience-trace/command-manifest.mjs", + "scripts/experience-trace/contract.mjs", + "scripts/experience-trace/index.mjs", + "scripts/experience-trace/project-source.mjs", "scripts/session-analysis/episode-facts.mjs", "scripts/session-analysis/result-facts.mjs", "scripts/session-analysis/session-core-facts.mjs", diff --git a/test/better-harness-cli.test.mjs b/test/better-harness-cli.test.mjs index e60a6f9..63c5bf8 100644 --- a/test/better-harness-cli.test.mjs +++ b/test/better-harness-cli.test.mjs @@ -340,6 +340,11 @@ test("better-harness CLI exposes command inventory as JSON", () => { harness.subcommands.find((subcommand) => subcommand.name === "source-review").script, "scripts/harness-analysis/report-source/cli.mjs", ); + assert.equal(harness.subcommands.find((subcommand) => subcommand.name === "experience-trace").audience, "maintainer"); + assert.equal( + harness.subcommands.find((subcommand) => subcommand.name === "experience-trace").script, + "scripts/experience-trace/cli.mjs", + ); assert.equal(harness.subcommands.some((subcommand) => subcommand.name === "prepare"), false); assert.equal(harness.subcommands.some((subcommand) => subcommand.name === "finalize"), false); assert.equal(harness.subcommands.some((subcommand) => subcommand.name === "apply-review"), false); @@ -713,7 +718,7 @@ test("better-harness CLI short-circuits help for every registered terminal path" // same dispatch (identical script, `invalid-before-help` stripped, trailing // `--help`); an identical dispatch yields byte-identical execution, so a // per-variant guarded subprocess would only re-run canonical's own args. - if (pathSegments.join(" ") === "harness source-review") continue; + if (["harness source-review", "harness experience-trace"].includes(pathSegments.join(" "))) continue; for (const helpFlag of ["--help", "-h"]) { const args = [...pathSegments, "invalid-before-help", helpFlag]; @@ -746,6 +751,21 @@ test("source-review help does not hide invalid phase or trailing arguments", () } }); +test("experience-trace owner validates trailing help arguments", () => { + for (const args of [ + ["harness", "experience-trace", "bogus", "--help"], + ["harness", "experience-trace", "--help", "trailing"], + ]) { + const dispatch = resolveDispatch(args); + assert.equal(dispatch.kind, "dispatch"); + assert.deepEqual(dispatch.args, args.slice(2)); + const result = runBetterHarness(args); + assert.equal(result.status, 64); + assert.equal(result.stdout, ""); + assert.equal(result.stderr, "INVALID_USAGE: invalid experience-trace arguments\n"); + } +}); + test("better-harness CLI preserves built-in discovery help and literal positional help", () => { const commands = runBetterHarness(["commands", "--help", "--audience", "advanced"]); assert.equal(commands.status, 0, commands.stderr); @@ -772,6 +792,7 @@ test("better-harness CLI group help projects workflow commands", () => { assert.equal(commands.includes("checkup"), true); assert.equal(commands.includes("record-fix-output"), true); assert.equal(commands.includes("source"), false); + assert.equal(commands.includes("experience-trace"), false); assert.equal(commands.includes("render"), false); }); @@ -790,6 +811,7 @@ test("better-harness CLI group help expands advanced and maintainer subcommands" const maintainerCommands = listedSubcommands(maintainer.stdout); assert.equal(maintainerCommands.includes("source"), true); assert.equal(maintainerCommands.includes("source-review"), true); + assert.equal(maintainerCommands.includes("experience-trace"), true); assert.equal(maintainerCommands.includes("repair-findings"), true); }); @@ -859,7 +881,7 @@ test("better-harness CLI emits a machine-readable schema", () => { }); test("registered harness commands remain executable through the Node facade", () => { - for (const subcommand of ["evidence-bundle", "analyze", "selection-profile", "source", "source-review", "task-loop-report", "render", "record-fix-output"]) { + for (const subcommand of ["evidence-bundle", "analyze", "selection-profile", "source", "source-review", "experience-trace", "task-loop-report", "render", "record-fix-output"]) { const result = runBetterHarness(["harness", subcommand, "--help"]); assert.equal(result.status, 0, `${subcommand}: ${result.stderr}`); assert.match(result.stdout, /Usage:/, subcommand); diff --git a/test/experience-trace-cli.test.mjs b/test/experience-trace-cli.test.mjs new file mode 100644 index 0000000..e29c690 --- /dev/null +++ b/test/experience-trace-cli.test.mjs @@ -0,0 +1,346 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; + +import { createHarnessReportSource } from "../scripts/harness-analysis/report-source.mjs"; +import { buildObservationManifest } from "../scripts/session-analysis/observation-manifest.mjs"; + +const ROOT = process.cwd(); +const LEAF = path.join(ROOT, "scripts", "experience-trace", "cli.mjs"); +const ROOT_CLI = path.join(ROOT, "scripts", "better-harness.mjs"); +const TASK_KEY = "task-key-00000001"; +const WORKSPACE_KEY = "workspace-key-00000001"; +const RUN_KEY = "run-key-00000001"; +const PRIVATE_SENTINEL = "private-experience-trace-sentinel"; +const ERROR_MESSAGES = Object.freeze({ + INVALID_USAGE: "invalid experience-trace arguments", + MISSING_EPISODE_SELECTION: "select exactly one episode mode", + INVALID_TRACE_BINDING: "trace binding key is invalid", + SOURCE_READ_FAILED: "unable to read report source", + TRACE_READ_FAILED: "unable to read experience trace", + TRACE_BOUNDS_EXCEEDED: "experience trace bounds exceeded", + INVALID_REPORT_SOURCE: "report source is invalid", + UNSUPPORTED_TRACE_SOURCE_VERSION: "report source version is unsupported", + UNSUPPORTED_TRACE_PLATFORM: "report source platform is unsupported", + UNKNOWN_EPISODE_REF: "selected episode is unavailable", + INVALID_EXPERIENCE_TRACE: "experience trace is invalid", +}); + +function run(entrypoint, args, options = {}) { + return spawnSync(process.execPath, [entrypoint, ...args], { + cwd: ROOT, + encoding: "utf8", + ...options, + }); +} + +function runLeaf(args, options) { + return run(LEAF, args, options); +} + +function runRoot(args, options) { + return run(ROOT_CLI, args, options); +} + +async function withTemporaryDirectory(runTest) { + const directory = await mkdtemp(path.join(os.tmpdir(), "better-harness-experience-trace-")); + try { + return await runTest(directory); + } finally { + await rm(directory, { recursive: true, force: true }); + } +} + +function validNoSessionSource({ warnings = [] } = {}) { + const manifest = buildObservationManifest({ + scope: { platform: "qoder", workspace: "fixture-workspace" }, + sources: [{ + id: "qoder-fixture", + kind: "project-jsonl", + enabled: true, + exists: true, + optional: false, + workspaceScoped: true, + }], + warnings: warnings.map((code) => ({ code })), + eligibleCount: 0, + analyzedCount: 0, + selectionStrategy: "all-eligible", + }); + manifest.sources.fingerprint = "0123456789abcdef"; + return createHarnessReportSource({ + manifest, + repositoryEvidence: {}, + sessionEvents: {}, + taskEpisodes: [], + deliveryEvidence: [], + semanticFacets: [], + interventionLedger: [], + evidenceRefs: [], + assessmentDecisions: [], + }); +} + +function createArgs(sourcePath, extra = []) { + return [ + "create", + "--source", sourcePath, + "--task-key", TASK_KEY, + "--workspace-key", WORKSPACE_KEY, + "--run-key", RUN_KEY, + "--no-session-evidence", + "--jsonl", + ...extra, + ]; +} + +function assertFailure(result, { code, exitCode = 1, privateValue } = {}) { + assert.equal(result.status, exitCode, result.stderr); + assert.equal(result.signal, null); + assert.equal(result.stdout, ""); + assert.equal(result.stderr, `${code}: ${ERROR_MESSAGES[code]}\n`); + if (privateValue) assert.doesNotMatch(`${result.stdout}${result.stderr}`, new RegExp(privateValue, "u")); +} + +test("create and validate round-trip through direct and root commands without writing artifacts", async () => { + await withTemporaryDirectory(async (directory) => { + const sourcePath = path.join(directory, "report.source.json"); + const tracePath = path.join(directory, "captured.trace.jsonl"); + await writeFile(sourcePath, JSON.stringify(validNoSessionSource())); + + const directCreate = runLeaf(createArgs(sourcePath)); + assert.equal(directCreate.status, 0, directCreate.stderr); + assert.equal(directCreate.signal, null); + assert.equal(directCreate.stderr, ""); + assert.equal(directCreate.stdout.endsWith("\n"), true); + assert.equal((await readdir(directory)).sort().join(","), "report.source.json"); + + const rootCreate = runRoot(["harness", "experience-trace", ...createArgs(sourcePath)]); + assert.equal(rootCreate.status, 0, rootCreate.stderr); + assert.equal(rootCreate.stderr, ""); + assert.equal(rootCreate.stdout, directCreate.stdout); + assert.equal((await readdir(directory)).sort().join(","), "report.source.json"); + + await writeFile(tracePath, directCreate.stdout); + const directValidate = runLeaf(["validate", "--trace", tracePath]); + const rootValidate = runRoot(["harness", "experience-trace", "validate", "--trace", tracePath]); + assert.equal(directValidate.status, 0, directValidate.stderr); + assert.equal(directValidate.stderr, ""); + assert.equal(rootValidate.status, 0, rootValidate.stderr); + assert.equal(rootValidate.stderr, ""); + assert.equal(rootValidate.stdout, directValidate.stdout); + const validation = JSON.parse(directValidate.stdout); + assert.deepEqual(Object.keys(validation), [ + "eventCount", "evidenceStatus", "kind", "recordCount", "schemaVersion", + "streamStatus", "traceDigest", "traceId", "valid", + ]); + assert.equal(validation.valid, true); + assert.equal(validation.evidenceStatus, "unavailable"); + assert.equal((await readdir(directory)).sort().join(","), "captured.trace.jsonl,report.source.json"); + }); +}); + +test("help is exact and strict parser failures do not read or echo private arguments", async () => { + await withTemporaryDirectory(async (directory) => { + const privatePath = path.join(directory, `${PRIVATE_SENTINEL}.json`); + for (const argv of [["--help"], ["-h"], ["create", "--help"], ["validate", "-h"]]) { + const result = runLeaf(argv); + assert.equal(result.status, 0, result.stderr); + assert.equal(result.stderr, ""); + assert.match(result.stdout, /^Better Harness Experience Trace v1/mu); + } + + const invalidCases = [ + ["bogus"], + ["create", "--help", "trailing"], + ["create", "--source"], + ["create", "--source", privatePath, "--task-key", TASK_KEY, "--workspace-key", WORKSPACE_KEY, "--run-key", RUN_KEY, "--no-session-evidence"], + [...createArgs(privatePath), "--source", privatePath], + [...createArgs(privatePath), "--no-session-evidence"], + ["create", "--source", privatePath, "--task-key", TASK_KEY, "--workspace-key", WORKSPACE_KEY, "--run-key", RUN_KEY, "--episode-ref", "episode:111111111111", "--no-session-evidence", "--jsonl"], + ["validate", "--trace", privatePath, "--unknown"], + ]; + for (const argv of invalidCases) { + assertFailure(runLeaf(argv), { code: "INVALID_USAGE", exitCode: 64, privateValue: PRIVATE_SENTINEL }); + } + + assertFailure(runLeaf(createArgs(privatePath)), { + code: "SOURCE_READ_FAILED", + privateValue: PRIVATE_SENTINEL, + }); + assertFailure(runLeaf(["validate", "--trace", privatePath]), { + code: "TRACE_READ_FAILED", + privateValue: PRIVATE_SENTINEL, + }); + }); +}); + +test("invalid source, private warning, deep trace, and oversized files fail before any stdout", async () => { + await withTemporaryDirectory(async (directory) => { + const sourcePath = path.join(directory, "source.json"); + const warningPath = path.join(directory, "warning.json"); + const unsupportedVersionPath = path.join(directory, "unsupported-version.json"); + const unsupportedPlatformPath = path.join(directory, "unsupported-platform.json"); + const invalidUtf8SourcePath = path.join(directory, "invalid-utf8.json"); + const invalidUtf8TracePath = path.join(directory, "invalid-utf8.jsonl"); + const deepTracePath = path.join(directory, "deep.jsonl"); + const oversizedTracePath = path.join(directory, "oversized.jsonl"); + const oversizedSourcePath = path.join(directory, "oversized-source.json"); + await writeFile(sourcePath, JSON.stringify(validNoSessionSource())); + await writeFile(warningPath, JSON.stringify(validNoSessionSource({ warnings: [PRIVATE_SENTINEL] }))); + const unsupportedVersion = validNoSessionSource(); + unsupportedVersion.schemaVersion = 4; + await writeFile(unsupportedVersionPath, JSON.stringify(unsupportedVersion)); + const unsupportedPlatform = validNoSessionSource(); + unsupportedPlatform.manifest.scope.platform = "codex"; + await writeFile(unsupportedPlatformPath, JSON.stringify(unsupportedPlatform)); + await writeFile(invalidUtf8SourcePath, Buffer.from([0xff])); + await writeFile(invalidUtf8TracePath, Buffer.from([0xff])); + + let deep = null; + for (let index = 0; index <= 64; index += 1) deep = { next: deep }; + await writeFile(deepTracePath, `${JSON.stringify(deep)}\n`); + await writeFile(oversizedTracePath, Buffer.alloc(1024 * 1024 + 1, 0x61)); + await writeFile(oversizedSourcePath, Buffer.alloc(16 * 1024 * 1024 + 1, 0x20)); + + assertFailure(runLeaf(createArgs(warningPath)), { + code: "INVALID_REPORT_SOURCE", + privateValue: PRIVATE_SENTINEL, + }); + assertFailure(runLeaf(createArgs(invalidUtf8SourcePath)), { code: "INVALID_REPORT_SOURCE" }); + assertFailure(runLeaf(["validate", "--trace", invalidUtf8TracePath]), { code: "INVALID_EXPERIENCE_TRACE" }); + assertFailure(runRoot(["harness", "experience-trace", "validate", "--trace", invalidUtf8TracePath]), { + code: "INVALID_EXPERIENCE_TRACE", + }); + assertFailure(runLeaf(["validate", "--trace", deepTracePath]), { code: "INVALID_EXPERIENCE_TRACE" }); + assertFailure(runLeaf(["validate", "--trace", oversizedTracePath]), { code: "TRACE_BOUNDS_EXCEEDED" }); + assertFailure(runLeaf(createArgs(oversizedSourcePath)), { code: "TRACE_BOUNDS_EXCEEDED" }); + + for (const [pathValue, code] of [ + [unsupportedVersionPath, "UNSUPPORTED_TRACE_SOURCE_VERSION"], + [unsupportedPlatformPath, "UNSUPPORTED_TRACE_PLATFORM"], + ]) { + assertFailure(runLeaf([ + "create", + "--source", pathValue, + "--task-key", TASK_KEY, + "--workspace-key", WORKSPACE_KEY, + "--run-key", RUN_KEY, + "--episode-ref", "episode:not-valid", + "--jsonl", + ]), { code }); + } + + const absentEpisode = runLeaf([ + "create", + "--source", sourcePath, + "--task-key", TASK_KEY, + "--workspace-key", WORKSPACE_KEY, + "--run-key", RUN_KEY, + "--episode-ref", "episode:111111111111", + "--jsonl", + ]); + assertFailure(absentEpisode, { code: "UNKNOWN_EPISODE_REF" }); + }); +}); + +test("shared line, record, and event bounds keep the public CLI error stable and private", async () => { + await withTemporaryDirectory(async (directory) => { + const overlongLinePath = path.join(directory, `overlong-line-${PRIVATE_SENTINEL}.jsonl`); + const tooManyRecordsPath = path.join(directory, `too-many-records-${PRIVATE_SENTINEL}.jsonl`); + const tooManyEventsPath = path.join(directory, `too-many-events-${PRIVATE_SENTINEL}.jsonl`); + + const invalidUtf8 = Buffer.from([0xff]); + const overlongUtf8Line = Buffer.concat([ + Buffer.from(JSON.stringify("界".repeat(21_846)), "utf8"), + invalidUtf8, + ]); + assert.equal(overlongUtf8Line.length > 65_536, true); + await writeFile(overlongLinePath, Buffer.concat([ + Buffer.from("{}\n"), + overlongUtf8Line, + Buffer.from("\n{}\n"), + ])); + await writeFile(tooManyRecordsPath, Buffer.concat([ + Buffer.from(Array.from({ length: 256 }, () => "{}\n").join("")), + invalidUtf8, + Buffer.from("\n"), + ])); + await writeFile(tooManyEventsPath, Buffer.concat([ + Buffer.from(`${[ + JSON.stringify({ kind: "better-harness.experience-trace.header" }), + ...Array.from({ length: 65 }, () => JSON.stringify({ kind: "better-harness.experience-trace.event" })), + ].join("\n")}\n`), + invalidUtf8, + Buffer.from("\n"), + ])); + + for (const tracePath of [overlongLinePath, tooManyRecordsPath, tooManyEventsPath]) { + for (const result of [ + runLeaf(["validate", "--trace", tracePath]), + runRoot(["harness", "experience-trace", "validate", "--trace", tracePath]), + ]) { + assertFailure(result, { + code: "TRACE_BOUNDS_EXCEEDED", + privateValue: PRIVATE_SENTINEL, + }); + } + } + }); +}); + +test("under-bound missing final LF remains invalid through direct and root validation", async () => { + await withTemporaryDirectory(async (directory) => { + const tracePath = path.join(directory, `missing-final-lf-${PRIVATE_SENTINEL}.jsonl`); + const fixture = await readFile(path.join( + ROOT, + "docs", + "specs", + "fixtures", + "lc03-no-session-v1.jsonl", + )); + assert.equal(fixture.at(-1), 0x0a); + await writeFile(tracePath, fixture.subarray(0, -1)); + + for (const result of [ + runLeaf(["validate", "--trace", tracePath]), + runRoot(["harness", "experience-trace", "validate", "--trace", tracePath]), + ]) { + assertFailure(result, { + code: "INVALID_EXPERIENCE_TRACE", + privateValue: PRIVATE_SENTINEL, + }); + } + }); +}); + +test("event bounds win when the terminal record is unterminated", async () => { + await withTemporaryDirectory(async (directory) => { + const tracePath = path.join(directory, `unterminated-event-overflow-${PRIVATE_SENTINEL}.jsonl`); + const trace = [ + JSON.stringify({ kind: "better-harness.experience-trace.header" }), + ...Array.from( + { length: 65 }, + () => JSON.stringify({ kind: "better-harness.experience-trace.event" }), + ), + JSON.stringify({ kind: "better-harness.experience-trace.terminal" }), + ].join("\n"); + assert.equal(trace.endsWith("\n"), false); + assert.equal(trace.match(/\n/gu)?.length, 66); + assert.equal(Buffer.byteLength(trace, "utf8") < 1024 * 1024, true); + await writeFile(tracePath, trace); + + for (const result of [ + runLeaf(["validate", "--trace", tracePath]), + runRoot(["harness", "experience-trace", "validate", "--trace", tracePath]), + ]) { + assertFailure(result, { + code: "TRACE_BOUNDS_EXCEEDED", + privateValue: PRIVATE_SENTINEL, + }); + } + }); +}); diff --git a/test/experience-trace-contract.test.mjs b/test/experience-trace-contract.test.mjs new file mode 100644 index 0000000..509e6df --- /dev/null +++ b/test/experience-trace-contract.test.mjs @@ -0,0 +1,271 @@ +import assert from "node:assert/strict"; +import { readFile } from "node:fs/promises"; +import test from "node:test"; + +import { + EXPERIENCE_TRACE_BOUNDS, + ExperienceTraceError, + assertIterativeJsonStructure, + bindingRefFromKey, + canonicalJson, + createTraceFromProjection, + experienceTraceValidationDocument, + parseAndValidateExperienceTraceJsonl, + serializeExperienceTrace, + traceDigestFor, + validateExperienceTraceRecords, +} from "../scripts/experience-trace/contract.mjs"; + +const noSessionFixtureUrl = new URL("../docs/specs/fixtures/lc03-no-session-v1.jsonl", import.meta.url); +const episodeFixtureUrl = new URL("../docs/specs/fixtures/lc03-episode-v1.jsonl", import.meta.url); + +async function fixtureText(url = noSessionFixtureUrl) { + return readFile(url, "utf8"); +} + +function canonicalJsonl(records) { + return records.map((record) => `${canonicalJson(record)}\n`).join(""); +} + +function noSessionProjection() { + return { + schemaVersion: 1, + producer: { platform: "qoder", reportSourceSchemaVersion: 3, manifestSchemaVersion: 2 }, + selection: { + sourceFingerprint: "0123456789abcdef", + strategy: "all-eligible", + eligibleCount: 0, + analyzedCount: 0, + sampled: false, + representative: true, + confidence: "Low", + warningCodes: [], + }, + episode: null, + absenceReason: "caller-declared-no-session-evidence", + }; +} + +function episodeProjection() { + return { + schemaVersion: 1, + producer: { platform: "qoder", reportSourceSchemaVersion: 3, manifestSchemaVersion: 2 }, + selection: { + sourceFingerprint: "fedcba9876543210", + strategy: "stratified", + eligibleCount: 3, + analyzedCount: 2, + sampled: true, + representative: false, + confidence: "Medium", + warningCodes: ["disabled-source-root", "missing-optional-root"], + }, + episode: { + episodeRef: "episode:abcdefabcdef", + sessionCount: 1, + continuation: "session-bounded", + startBoundary: "session-start", + toolCallCount: 5, + changeSets: [{ eventCount: 2, firstOrdinal: 3, lastOrdinal: 4, targetKeys: ["11111111111111111111", "22222222222222222222"] }], + validationSets: [{ + category: "node --test", + status: "passed", + sourceOrdinal: 7, + checkIdentity: "check:aaaaaaaaaaaaaaaaaaaaaaaa", + targetKeys: ["33333333333333333333"], + }], + permissionBoundary: { prompted: 1, denied: 0, escalated: 0, protectedActions: 1 }, + closureStatus: "closed", + repairStatus: "repaired-and-passed", + }, + absenceReason: null, + }; +} + +function assertInvalid(callback) { + assert.throws(callback, (error) => error instanceof ExperienceTraceError && error.code === "INVALID_EXPERIENCE_TRACE"); +} + +function assertCode(callback, code) { + assert.throws(callback, (error) => error instanceof ExperienceTraceError && error.code === code); +} + +test("consumes the normative no-session fixture and reproduces every golden vector", async () => { + const fixture = await fixtureText(); + const records = parseAndValidateExperienceTraceJsonl(fixture); + assert.equal(records.length, 12); + assert.equal(serializeExperienceTrace(records), fixture); + assert.equal(records[0].source.projectionDigest, "sha256:f46c5aaea639376da6fca7bfa9df215ee2a22bdad509761912a705d57d8eb9c9"); + assert.equal(records[0].traceId, "trace:sha256:a924fb792337d1a641f2cc152a67764387d6ae0b0d04fa6ff0752469ab8d095f"); + assert.equal(records.at(-1).traceDigest, "sha256:e568040d4ce087a63abb593e760ea4b94b140a8da6421dad7b69ab583426956e"); + assert.deepEqual(experienceTraceValidationDocument(records), { + kind: "better-harness.experience-trace.validation", + schemaVersion: 1, + valid: true, + traceId: records[0].traceId, + streamStatus: "complete", + evidenceStatus: "unavailable", + recordCount: 12, + eventCount: 10, + traceDigest: records.at(-1).traceDigest, + }); +}); + +test("creates the normative no-session byte stream without emitting raw caller keys", async () => { + const result = createTraceFromProjection({ + projection: noSessionProjection(), + taskKey: "task-key-00000001", + workspaceKey: "workspace-key-00000001", + runKey: "run-key-00000001", + }); + assert.equal(result.jsonl, await fixtureText()); + assert.equal(result.jsonl.includes("task-key-00000001"), false); + assert.equal(result.records[0].binding.episode.ref, null); + assert.equal(bindingRefFromKey("task", "task-key-00000001"), "task:sha256:fce82f8f9ff62cd9af044e2e21bfd2a1822d4cc506bf174a285b8fd03fb011e9"); + assert.equal(bindingRefFromKey("workspace", "workspace-key-00000001"), "workspace:sha256:50c688560321b967e32f410e861cf64cc7af45c817bddbe466aa1f703a29850e"); + assert.equal(bindingRefFromKey("run", "run-key-00000001"), "run:sha256:7522f027d7bb67cbdbb3818345249dc1f4429fe8e484f44c6330083f1292e1d0"); +}); + +test("reproduces the versioned Episode golden fixture and vectors byte-for-byte", async () => { + const projection = episodeProjection(); + const created = createTraceFromProjection({ + projection, + taskKey: "episode-task-key-01", + workspaceKey: "episode-workspace-01", + runKey: "episode-run-key-0001", + }); + const fixture = await fixtureText(episodeFixtureUrl); + assert.equal(created.jsonl, fixture); + const parsed = parseAndValidateExperienceTraceJsonl(created.jsonl); + assert.equal(parsed[0].source.projectionDigest, "sha256:670e6779abb62b2dae18d2f5081a11b102bbef3a9baa0799b1828d0666b4ff8b"); + assert.equal(parsed[0].traceId, "trace:sha256:4064d02402929c3629fdbeefa261106c0e9806c8802dcfbc5302ee2dae5279ee"); + assert.equal(parsed.at(-1).traceDigest, "sha256:f15892bcafa1ad3ac34586a0bdae412d73e995f84bd72ebe340b5470cf5ae5f1"); + assert.equal(parsed.at(-1).evidenceStatus, "partial"); + assert.equal(parsed.at(-1).eventCount, 11); + assert.equal(parsed.at(-1).recordCount, 13); + assert.equal(parsed[3].evidenceRef, "source:validation:1"); + assert.equal(serializeExperienceTrace(parsed), created.jsonl); +}); + +test("rejects an Episode header ref that disagrees with reconstructed event evidence", async () => { + const records = parseAndValidateExperienceTraceJsonl(await fixtureText(episodeFixtureUrl)); + const mismatched = structuredClone(records); + mismatched[0].binding.episode.ref = "episode:bbbbbbbbbbbb"; + mismatched.at(-1).traceDigest = traceDigestFor(mismatched.slice(0, -1)); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(canonicalJsonl(mismatched))); +}); + +test("rejects shared line, record, event, and byte bounds with the bounds code before JSON parsing", () => { + const tooManyRecords = `${Array(EXPERIENCE_TRACE_BOUNDS.maxRecords + 1).fill("{").join("\n")}\n`; + assertCode(() => parseAndValidateExperienceTraceJsonl(tooManyRecords), "TRACE_BOUNDS_EXCEEDED"); + + const tooManyEvents = `${Array(EXPERIENCE_TRACE_BOUNDS.maxEvents + 3).fill("{").join("\n")}\n`; + assertCode(() => parseAndValidateExperienceTraceJsonl(tooManyEvents), "TRACE_BOUNDS_EXCEEDED"); + + const oversizedLine = `${JSON.stringify("x".repeat(EXPERIENCE_TRACE_BOUNDS.maxLineBytes))}\n{}\n{}\n`; + assertCode(() => parseAndValidateExperienceTraceJsonl(oversizedLine), "TRACE_BOUNDS_EXCEEDED"); + + const exactLine = `${JSON.stringify("x".repeat(EXPERIENCE_TRACE_BOUNDS.maxLineBytes - 3))}\n{}\n{}\n`; + assertInvalid(() => parseAndValidateExperienceTraceJsonl(exactLine)); + + const oversizedTrace = Buffer.alloc(EXPERIENCE_TRACE_BOUNDS.maxTraceBytes + 1, 0x61); + assertCode(() => parseAndValidateExperienceTraceJsonl(oversizedTrace), "TRACE_BOUNDS_EXCEEDED"); + + assertCode( + () => validateExperienceTraceRecords(Array(EXPERIENCE_TRACE_BOUNDS.maxRecords + 1).fill(null)), + "TRACE_BOUNDS_EXCEEDED", + ); + assertCode( + () => validateExperienceTraceRecords(Array(EXPERIENCE_TRACE_BOUNDS.maxEvents + 3).fill(null)), + "TRACE_BOUNDS_EXCEEDED", + ); +}); + +test("raw-byte shared bounds outrank invalid UTF-8 without masking under-bound UTF-8 failures", () => { + const overTotal = Buffer.alloc(EXPERIENCE_TRACE_BOUNDS.maxTraceBytes + 1, 0x61); + overTotal[0] = 0xff; + assertCode(() => parseAndValidateExperienceTraceJsonl(overTotal), "TRACE_BOUNDS_EXCEEDED"); + + const overLine = Buffer.concat([ + Buffer.from([0xff]), + Buffer.alloc(EXPERIENCE_TRACE_BOUNDS.maxLineBytes, 0x61), + Buffer.from("\n"), + ]); + assertCode(() => parseAndValidateExperienceTraceJsonl(overLine), "TRACE_BOUNDS_EXCEEDED"); + + const overRecords = Buffer.concat([ + Buffer.from([0xff]), + Buffer.from("null\n".repeat(EXPERIENCE_TRACE_BOUNDS.maxRecords + 1)), + ]); + assertCode(() => parseAndValidateExperienceTraceJsonl(overRecords), "TRACE_BOUNDS_EXCEEDED"); + + const overEvents = new Uint8Array(Buffer.concat([ + Buffer.from([0xff]), + Buffer.from("null\n".repeat(EXPERIENCE_TRACE_BOUNDS.maxEvents + 3)), + ])); + assertCode(() => parseAndValidateExperienceTraceJsonl(overEvents), "TRACE_BOUNDS_EXCEEDED"); + + const underBoundInvalidUtf8 = Buffer.from([0xff, 0x0a, 0x7b, 0x7d, 0x0a, 0x7b, 0x7d, 0x0a]); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(underBoundInvalidUtf8)); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(`\ud800${"x".repeat(EXPERIENCE_TRACE_BOUNDS.maxLineBytes)}`)); + + // Sixty-six terminated lines model the header plus 65 events; the nonempty + // trailing segment is the unterminated terminal and must count as line 67. + const unterminatedOverEvents = Buffer.concat([ + Buffer.from("null\n".repeat(EXPERIENCE_TRACE_BOUNDS.maxEvents + 2)), + Buffer.from([0xff]), + ]); + assertCode(() => parseAndValidateExperienceTraceJsonl(unterminatedOverEvents), "TRACE_BOUNDS_EXCEEDED"); + + assertInvalid(() => parseAndValidateExperienceTraceJsonl("{}\n{}\n{}")); +}); + +test("rejects non-canonical line endings, BOMs, whitespace, duplicate keys, and reordered data", async () => { + const fixture = await fixtureText(); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(fixture.replaceAll("\n", "\r\n"))); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(`\uFEFF${fixture}`)); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(fixture.replace("{\"binding\"", "{ \"binding\""))); + assertInvalid(() => parseAndValidateExperienceTraceJsonl(fixture.replace("\"kind\":\"better-harness.experience-trace.header\"", "\"kind\":\"better-harness.experience-trace.header\",\"kind\":\"better-harness.experience-trace.header\""))); + + const lines = fixture.trimEnd().split("\n"); + [lines[1], lines[2]] = [lines[2], lines[1]]; + assertInvalid(() => parseAndValidateExperienceTraceJsonl(`${lines.join("\n")}\n`)); +}); + +test("rejects unknown fields and hash tampering even when the bytes are canonical", async () => { + const records = parseAndValidateExperienceTraceJsonl(await fixtureText()); + const unknown = structuredClone(records); + unknown[0].unexpected = true; + assertInvalid(() => parseAndValidateExperienceTraceJsonl(canonicalJsonl(unknown))); + + const tampered = structuredClone(records); + tampered.at(-1).traceDigest = `sha256:${"0".repeat(64)}`; + assertInvalid(() => parseAndValidateExperienceTraceJsonl(canonicalJsonl(tampered))); + + const reversedHeader = Object.fromEntries(Object.entries(records[0]).reverse()); + const text = `${JSON.stringify(reversedHeader)}\n${canonicalJsonl(records.slice(1))}`; + assertInvalid(() => parseAndValidateExperienceTraceJsonl(text)); +}); + +test("normalizes NFC while rejecting unsafe JSON strings and numbers", () => { + assert.equal(canonicalJson({ text: "e\u0301" }), "{\"text\":\"é\"}"); + assertInvalid(() => canonicalJson("\ud800")); + assertInvalid(() => canonicalJson(-0)); + assertInvalid(() => canonicalJson(1.5)); + assertInvalid(() => canonicalJson(Number.MAX_SAFE_INTEGER + 1)); +}); + +test("preflights deep JSON structures iteratively before recursive serialization", () => { + let allowed = null; + for (let index = 0; index < 64; index += 1) allowed = { next: allowed }; + assert.ok(Buffer.byteLength(JSON.stringify(allowed), "utf8") < 64 * 1024); + assert.equal(assertIterativeJsonStructure(allowed), 65); + assert.doesNotThrow(() => canonicalJson(allowed)); + + let rejected = null; + for (let index = 0; index < 65; index += 1) rejected = { next: rejected }; + assertInvalid(() => assertIterativeJsonStructure(rejected)); + assertInvalid(() => canonicalJson(rejected)); + + assertInvalid(() => assertIterativeJsonStructure(Array(EXPERIENCE_TRACE_BOUNDS.maxValues).fill(null))); +}); diff --git a/test/experience-trace.test.mjs b/test/experience-trace.test.mjs new file mode 100644 index 0000000..36433b1 --- /dev/null +++ b/test/experience-trace.test.mjs @@ -0,0 +1,294 @@ +import assert from "node:assert/strict"; +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; + +import { createHarnessReportSource } from "../scripts/harness-analysis/report-source.mjs"; +import { buildObservationManifest } from "../scripts/session-analysis/observation-manifest.mjs"; +import { + createExperienceTrace, + projectQoderReportSource, +} from "../scripts/experience-trace/project-source.mjs"; +import { + experienceTraceValidationDocument, + parseAndValidateExperienceTraceJsonl, +} from "../scripts/experience-trace/contract.mjs"; + +const TEST_DIR = path.dirname(fileURLToPath(import.meta.url)); +const NO_SESSION_FIXTURE = path.join(TEST_DIR, "..", "docs", "specs", "fixtures", "lc03-no-session-v1.jsonl"); +const EPISODE_REF = "episode:111111111111"; +const TASK_KEY = "task-key-00000001"; +const WORKSPACE_KEY = "workspace-key-00000001"; +const RUN_KEY = "run-key-00000001"; +const PRIVATE_SENTINEL = "private-trace-sentinel-do-not-emit"; + +function source({ + noSession = false, + warnings = ["missing-optional-root"], + validationSets = [ + { + category: "node --test", + status: "passed", + ordinal: 7, + checkIdentity: "check:111111111111111111111111", + targetKeys: ["aaaaaaaaaaaaaaaaaaaa"], + }, + ], +} = {}) { + const manifest = buildObservationManifest({ + scope: { platform: "qoder", workspace: "/workspace/private-project" }, + sources: [{ + id: "qoder-fixture", + kind: "project-jsonl", + enabled: true, + exists: true, + optional: false, + workspaceScoped: true, + }], + warnings: warnings.map((code) => ({ code })), + eligibleCount: noSession ? 0 : 10, + analyzedCount: noSession ? 0 : 10, + selectionStrategy: "all-eligible", + }); + manifest.sources.fingerprint = "0123456789abcdef"; + + return createHarnessReportSource({ + manifest, + repositoryEvidence: {}, + sessionEvents: noSession ? {} : { + permissionSummary: { + observed: 1, + routineAllowed: 0, + prompted: 1, + denied: 0, + escalated: 0, + protectedActions: 1, + }, + }, + taskEpisodes: noSession ? [] : [{ + id: EPISODE_REF, + sessionCount: 1, + continuation: "session-bounded", + startBoundary: "session-start", + toolCalls: 3, + changeSets: [{ + eventCount: 2, + firstOrdinal: 1, + lastOrdinal: 3, + targetKeys: ["bbbbbbbbbbbbbbbbbbbb", "aaaaaaaaaaaaaaaaaaaa"], + }], + validationSets, + permissionSummary: { + prompted: 1, + denied: 0, + escalated: 0, + protectedActions: 1, + evidenceRefs: [{ kind: "fixture", id: "permission-private-path" }], + }, + closure: { status: "closed", evidenceRefs: [] }, + repair: { status: "review-required", evidenceRefs: [] }, + }], + deliveryEvidence: [], + semanticFacets: [], + interventionLedger: [], + evidenceRefs: [], + assessmentDecisions: [], + }); +} + +function create(sourceValue, options = {}) { + return createExperienceTrace(sourceValue, { + taskKey: TASK_KEY, + workspaceKey: WORKSPACE_KEY, + runKey: RUN_KEY, + episodeRef: EPISODE_REF, + ...options, + }); +} + +function assertCode(callback, code) { + assert.throws(callback, (error) => error?.code === code && error.message === code); +} + +test("Episode projection is deterministic, canonical, and round-trips through the contract", () => { + const reportSource = source(); + const first = create(reportSource); + const second = create(reportSource); + + assert.equal(first.jsonl, second.jsonl); + assert.deepEqual(parseAndValidateExperienceTraceJsonl(first.jsonl), first.records); + assert.deepEqual(experienceTraceValidationDocument(first.records), { + kind: "better-harness.experience-trace.validation", + schemaVersion: 1, + valid: true, + traceId: first.records[0].traceId, + streamStatus: "complete", + evidenceStatus: "partial", + recordCount: first.records.length, + eventCount: first.records.length - 2, + traceDigest: first.records.at(-1).traceDigest, + }); + assert.equal(first.records[0].binding.task.provenance, "caller-asserted"); + assert.equal(first.records[0].binding.episode.ref, EPISODE_REF); +}); + +test("explicit no-session output matches the normative JSONL fixture", async () => { + const reportSource = source({ noSession: true, warnings: [] }); + const result = createExperienceTrace(reportSource, { + taskKey: TASK_KEY, + workspaceKey: WORKSPACE_KEY, + runKey: RUN_KEY, + noSessionEvidence: true, + }); + const fixture = await readFile(NO_SESSION_FIXTURE, "utf8"); + + assert.equal(result.jsonl, fixture); + assert.equal(result.records.at(-1).evidenceStatus, "unavailable"); + assert.equal(result.records.filter((record) => record.kind.endsWith(".event")).length, 10); +}); + +test("no-session mode ignores unrelated retained Episodes instead of guessing one", () => { + const result = createExperienceTrace(source(), { + taskKey: TASK_KEY, + workspaceKey: WORKSPACE_KEY, + runKey: RUN_KEY, + noSessionEvidence: true, + }); + + assert.equal(result.records[0].binding.episode.ref, null); + assert.equal(result.records.at(-1).evidenceStatus, "unavailable"); + assert.doesNotMatch(result.jsonl, new RegExp(EPISODE_REF, "u")); + assert.equal(result.records.filter((record) => record.eventType === "task-episode").length, 0); +}); + +test("validation references follow normalized ordinal order and sets canonicalize", () => { + const reportSource = source({ + warnings: ["partial-secret-scan-coverage", "missing-optional-root", "partial-secret-scan-coverage"], + validationSets: [ + { + category: "npm test", + status: "failed", + ordinal: 9, + checkIdentity: "check:999999999999999999999999", + targetKeys: ["ffffffffffffffffffff", "aaaaaaaaaaaaaaaaaaaa", "ffffffffffffffffffff"], + }, + { + category: "node --test", + status: "passed", + ordinal: 2, + checkIdentity: "check:222222222222222222222222", + targetKeys: ["cccccccccccccccccccc", "bbbbbbbbbbbbbbbbbbbb"], + }, + ], + }); + const result = create(reportSource); + const validationEvents = result.records.filter((record) => record.eventType === "validation-observation"); + + assert.deepEqual(validationEvents.map((record) => [record.evidenceRef, record.payload.sourceOrdinal]), [ + ["source:validation:1", 2], + ["source:validation:2", 9], + ]); + assert.deepEqual(validationEvents[1].payload.targetKeys, ["aaaaaaaaaaaaaaaaaaaa", "ffffffffffffffffffff"]); + assert.deepEqual(result.records[0].selection.warningCodes, [ + "missing-optional-root", + "partial-secret-scan-coverage", + ]); +}); + +test("marker failures use the documented combined-failure precedence", () => { + const invalidKind = source(); + invalidKind.schemaVersion = 99; + invalidKind.kind = "not-a-report-source"; + invalidKind.manifest.scope.platform = "codex"; + assertCode(() => projectQoderReportSource(invalidKind, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); + + const unsupportedVersion = source(); + unsupportedVersion.schemaVersion = 99; + unsupportedVersion.manifest.scope.platform = "codex"; + assertCode(() => projectQoderReportSource(unsupportedVersion, { episodeRef: EPISODE_REF }), "UNSUPPORTED_TRACE_SOURCE_VERSION"); + + const unsupportedPlatform = source(); + unsupportedPlatform.manifest.scope.platform = "codex"; + assertCode(() => projectQoderReportSource(unsupportedPlatform, { episodeRef: EPISODE_REF }), "UNSUPPORTED_TRACE_PLATFORM"); +}); + +test("source markers take precedence over direct-call selection errors", () => { + const unsupportedVersion = source(); + unsupportedVersion.schemaVersion = 99; + + assertCode( + () => projectQoderReportSource(unsupportedVersion, { episodeRef: "not-an-episode-ref" }), + "UNSUPPORTED_TRACE_SOURCE_VERSION", + ); +}); + +test("unknown warnings and invalid projected fields fail without leaking their values", () => { + const unknownWarning = source({ warnings: [PRIVATE_SENTINEL] }); + assert.throws( + () => projectQoderReportSource(unknownWarning, { episodeRef: EPISODE_REF }), + (error) => error?.code === "INVALID_REPORT_SOURCE" && !error.message.includes(PRIVATE_SENTINEL), + ); + + const missingToolCalls = source(); + delete missingToolCalls.taskEpisodes[0].toolCalls; + assertCode(() => projectQoderReportSource(missingToolCalls, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); + + const tooManyChanges = source(); + tooManyChanges.taskEpisodes[0].changeSets = Array.from({ length: 17 }, () => ({ + eventCount: 0, + firstOrdinal: 0, + lastOrdinal: 0, + targetKeys: [], + })); + assertCode(() => projectQoderReportSource(tooManyChanges, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); + + const tooManyTargets = source(); + tooManyTargets.taskEpisodes[0].changeSets[0].targetKeys = Array.from( + { length: 13 }, + (_, index) => `${index.toString(16).padStart(20, "0")}`, + ); + assertCode(() => projectQoderReportSource(tooManyTargets, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); + + const duplicateOrdinal = source({ + validationSets: [ + { + category: "node --test", + status: "passed", + ordinal: 1, + checkIdentity: "check:111111111111111111111111", + targetKeys: [], + }, + { + category: "npm test", + status: "failed", + ordinal: 1, + checkIdentity: "check:222222222222222222222222", + targetKeys: [], + }, + ], + }); + assertCode(() => projectQoderReportSource(duplicateOrdinal, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); +}); + +test("unrelated nested source content never enters an allowlist projection", () => { + const reportSource = source(); + reportSource.taskEpisodes[0].unrelatedDebug = { + opaquePrivateValue: PRIVATE_SENTINEL, + nested: { value: PRIVATE_SENTINEL }, + }; + + const result = create(reportSource); + assert.doesNotMatch(result.jsonl, new RegExp(PRIVATE_SENTINEL, "u")); + assert.doesNotMatch(JSON.stringify(result.records), /permission-private-path/u); +}); + +test("deep source input fails at the iterative preflight boundary", () => { + const reportSource = source(); + let cursor = reportSource.taskEpisodes[0]; + for (let index = 0; index <= 64; index += 1) { + cursor.nested = {}; + cursor = cursor.nested; + } + assertCode(() => projectQoderReportSource(reportSource, { episodeRef: EPISODE_REF }), "INVALID_REPORT_SOURCE"); +}); diff --git a/test/fixtures/scripts-refactor-contract/harness-help.txt b/test/fixtures/scripts-refactor-contract/harness-help.txt index 1d0148b..e9ac0a2 100644 --- a/test/fixtures/scripts-refactor-contract/harness-help.txt +++ b/test/fixtures/scripts-refactor-contract/harness-help.txt @@ -15,6 +15,7 @@ Subcommands: selection-profile Build a privacy-safe session selection profile source Build the deterministic Agent Work Loop source envelope source-review Create, compile, and apply a bounded report-source review + experience-trace Create and validate task-scoped, privacy-safe Experience Trace JSONL task-loop-report Project deterministic task-loop findings from report.source.json render Render reviewed findings data into report artifacts preview-canvas Preview a Qoder Canvas report on this machine diff --git a/test/fixtures/scripts-refactor-contract/root-help.txt b/test/fixtures/scripts-refactor-contract/root-help.txt index 4171075..267968d 100644 --- a/test/fixtures/scripts-refactor-contract/root-help.txt +++ b/test/fixtures/scripts-refactor-contract/root-help.txt @@ -17,9 +17,9 @@ Commands: Workflows harness Check Better Harness readiness reports and Canvas outputs subcommands: evidence-bundle, workspace-topology, analyze, checkup, - selection-profile, source, source-review, task-loop-report, render, - preview-canvas, report-quality, repair-findings, record-fix-output, - validate-canvas + selection-profile, source, source-review, experience-trace, + task-loop-report, render, preview-canvas, report-quality, + repair-findings, record-fix-output, validate-canvas Project Evidence session-analysis Collect and normalize Qoder, Codex, Claude, Cursor, Qwen, Copilot, diff --git a/test/scripts-refactor-contract.test.mjs b/test/scripts-refactor-contract.test.mjs index 96772a7..3b779ec 100644 --- a/test/scripts-refactor-contract.test.mjs +++ b/test/scripts-refactor-contract.test.mjs @@ -107,17 +107,17 @@ test("scripts refactor contract freezes machine-readable CLI output", () => { { label: "command inventory", args: ["commands", "--json"], - sha256: "c5fb49bd619ba8337b103b33db7f205592ca55b956686fe53466604f697087aa", + sha256: "b6896996a90add07c109bb58032d14ed2d25105eaf86eb96253ad742c6ee865b", }, { label: "OpenCLI schema", args: ["schema"], - sha256: "0283010f3ce112ab6b755ca1e0ea01cb76fc073d9fb2f8ddeeb4a64520459a2c", + sha256: "ec857f3a0d40aa34bdc20c4cbba30781a60932e8997375190abde015b0732c33", }, { label: "Harness command description", args: ["command", "describe", "harness", "--json"], - sha256: "aa7aeb8d28360da25a83dc65b9fea583b9abf7ae2e665c307cf6c838937dd1d4", + sha256: "38fba2f466e79aa7ca458a66aa9859bc4b748b5ae2684efc36b948ab61c6bfd0", }, ];