Spec: validator-facing eval interface — what SN74 consumes from LoopOver as the objective eval provider

[JSONbored]

Context

LoopOver's verification stack (ledger + anchor + published decision records + deterministic replay + certified precision-at-coverage) currently proves claims about our own pipeline's decisions. Nothing exposes scores in a form an SN74 validator could consume for weight-setting. The subnet is moving toward objective optimization, and LoopOver has been invited to be its eval/scoring substrate — this spec converts that from an informal signal into a concrete mechanism proposal. It is the prerequisite for the frozen-repo benchmark harness (do not decompose that before this closes).

Requirements — each gets an explicit recorded decision

1. The consumable artifact: the per-(agent, work-unit) score record — exact shape, what it commits to (corpus/window, scoring-rule version, attestation reference), how it is signed, freshness semantics.
2. Transport: pull API vs published signed bundles vs on-chain commitment (the SN74-hotkey anchor path is the natural candidate for the commitment variant) — decide the v1 and the upgrade path.
3. Scoring semantics offered: outcome-confirmed precision-at-coverage; benchmark-run scores (future frozen-repo harness); abstention/coverage rules; and an explicit list of what we do not score.
4. Trust model per artifact, tied to the cloud: unified tenant verification contract — claim → artifact → endpoint → trust assumption #9186 verification matrix: attested vs reproducible vs asserted — a validator must be able to see which tier every number sits in.
5. Governance of the scoring rules themselves: changes to scoring gate through the existing prospective-backtest machinery (Pareto floor, held-out splits), with versioning and an announcement contract.
6. What the subnet side runs vs trusts: the verifier CLI, the contract docs, and the minimum a validator must execute to consume scores trustlessly.

Hard constraints

• ⚠️ No emissions, pricing, or weight economics in this issue or the resulting public doc — mechanism only. Economic terms are a private conversation.

Deliverables

• Decision-record comments resolving every numbered item, plus a draft docs page (not published until the mechanism conversation with the subnet team concludes).

Expected outcome

"Adopt LoopOver as the objective eval provider" becomes a reviewable mechanism proposal a validator team can evaluate line by line, rather than a vibe.

References

Relates to #6101, #9186, #8534.

[linear-code]

JSO-6389

[JSONbored]

Decision records — validator-facing eval interface

Resolving all six numbered requirements. Every shape below is grounded in what actually ships today; where something does not exist yet it is named as not-existing rather than specified as if it did.

Scope guard honored: no emissions, pricing, or weight economics appear here or in the resulting doc. Mechanism only.

---

1. The consumable artifact — EvalScoreRecord

Decision: one record type, per (subject, workUnit) pair, that commits to its own inputs by digest.

type EvalScoreRecord = {
  schemaVersion: 1;

  /** WHO was scored. Opaque to LoopOver — an SS58 address, a pubkey, whatever the consumer
   *  keys on. LoopOver never interprets it, so identity policy stays entirely subnet-side. */
  subject: { kind: "agent"; id: string };

  /** WHAT was scored. `outcome_confirmed_precision` ships today (§3a);
   *  `benchmark_run` is the frozen-repo harness (#9216) and MUST NOT appear until that lands. */
  workUnit:
    | { kind: "outcome_confirmed_precision"; ruleId: string }
    | { kind: "benchmark_run"; benchmarkId: string; snapshotRef: string };

  /** THE NUMBERS. `null` is never `0` — below the sample floor a metric is absent, not zero.
   *  Mirrors the existing PublicRulePrecisionRow discipline exactly. */
  score: {
    decided: number;
    confirmed: number;
    precision: number | null;
    recall: number | null;
    coverage: number | null;
    abstained: number;
  };

  /** WHAT THE NUMBERS COMMIT TO. Without this block a score is unfalsifiable. */
  commitments: {
    corpusChecksum: string;      // BacktestCorpusManifest.checksum (sha256, key-sorted canonical)
    scoringRuleVersion: string;  // §5 — semantics version, not the package version
    windowStart: string;
    windowEnd: string;
    splitSeed: string | null;    // splitBacktestCorpus seed; null when no split applies
    heldOutFraction: number | null;
  };

  /** WHICH TIER (§4). `attestation` present iff tier === "attested". */
  trust: {
    tier: "attested" | "reproducible" | "asserted";
    attestation?: { envelopeDigest: string; measurement: string; runId: string };
  };

  issuedAt: string;
  /** contentDigest(record-without-this-field) — same canonicalJson + sha256 primitives as
   *  DecisionRecord (src/review/decision-record.ts). Recomputable by the consumer. */
  recordDigest: string;
};

Signing — the honest answer: not signed in v1, and that is a deliberate decision, not an oversight. No artifact-signing infrastructure exists in this repo today. The only signing that ships is signFederatedBundle (src/orb/federated-bundle.ts:188), an HMAC over an operator-managed peer allowlist — explicitly not PKI, and unsuitable for an open validator audience where the verifier does not pre-share a secret with us. buildLedgerAnchorPayload (src/review/decision-record.ts:376) exists but its own doc comment states it is called from nowhere.

A signature we can mint is also a signature we can re-mint over rewritten history, so it would add ceremony without adding a guarantee against the operator — the one adversary a validator actually cares about. v1 therefore commits by digest + independent recomputation (the consumer re-derives every number from the checksummed corpus and public scoring code), and §2's upgrade path is where a signature starts to mean something.

Freshness: windowEnd + issuedAt are the freshness signal; consumers set their own staleness policy. Records are immutable — a corrected score is a new record with a later issuedAt, never a mutation, matching how supersessions are visible history in the decision ledger rather than silent replacement.

---

2. Transport

Decision: v1 is a pull API. Signed bundles and on-chain commitment are the upgrade path, in that order.

Option	v1?	Why
Pull API	✅ v1	Extends the existing, working /v1/public/* surface (/v1/public/stats, /v1/public/decision-ledger/verify, /v1/public/decision-records/:owner/:repo/:pull — all unauthenticated, all shipped). Zero new trust machinery; a validator polls and recomputes.
Published signed bundles	upgrade	Needs a real key-distribution story (see §1) — not just a signing call. Worth doing when consumers need offline/batch verification without reaching our endpoint.
On-chain commitment	upgrade, later	The natural anchor for the tamper-evident→tamper-proof gap (migrations/0180_decision_ledger.sql:9-12). Highest value, highest coordination cost; the SN74-hotkey anchor path is the obvious candidate.

v1 endpoint: GET /v1/public/eval-scores?subject=<id>&since=<iso> → { records: EvalScoreRecord[] }, unauthenticated, same cache posture as /v1/public/stats (public, max-age=60, stale-while-revalidate=300).

Upgrade path is additive, not a migration: recordDigest is already the exact preimage a future signature or on-chain commitment would sign over. Adding either later changes no existing field.

---

3. Scoring semantics offered

(a) Outcome-confirmed precision-at-coverage — available now. Already computed and published: loadPublicRulePrecision (src/review/public-rule-precision.ts:47) over a 90-day window with a 10-decided floor. Ground truth is recorded human-override events (signal.human_override:* in audit_events), with reversal provenance (reversal_reopened / reversal_reverted / reversal_superseded). Positive class is reversed — a prediction that the rule's original firing was wrong (backtest-score.ts:11-13).

(b) Benchmark-run scores — #9216, does not exist yet. Reuses scoreBacktest / compareBacktestScores unchanged. Must not be emitted until that harness lands; a record with workUnit.kind === "benchmark_run" before then would be asserting a number nothing produced.

(c) Abstention and coverage. Abstention is first-class, never silently folded into an error: coverage = decided / (decided + abstained). A high-precision/low-coverage agent and a broad/noisier one are distinguishable rather than collapsed into one figure. Below-floor metrics are null, never 0.

(d) Explicitly NOT scored — permanent list:

• Live gate decisions. Deliberately out of scope for attestation (Research: the attested-evaluation epic covers offline backtests only — decide whether the LIVE decision path needs attestation, and in what order #9141) and stated permanently in verify-this-review.mdx. Scores describe offline replay only.
• Subjective review quality. No "was this a good review" number. Nothing here scores taste.
• Data provenance of ground truth (T3, Research: is TEE hardware attestation necessary, or does a reproducible-backtest checksum approach cover the real threat? #8136). The override history is recorded on maintainer infrastructure. Attestation proves computation, not that the labels are honest — a perfectly attested run over cherry-picked labels is still cherry-picked. No tier offered here closes T3, and any consumer treating an attested tier as closing it has misread the guarantee.
• Anything requiring a private corpus until per-tenant export (cloud: tenant-verifiable backtests for private corpora — per-tenant checksummed corpus export #8539) lands.

---

4. Trust model per artifact

Decision: three tiers, on every record, tied to #9186's matrix.

Tier	Means	Verified by	Assumes
attested	Computation ran inside genuine SEV-SNP hardware; report chains to AMD's root	scripts/verify-attested-run.ts — offline, no LoopOver contact, exit code per failure class	AMD's root key; not ground-truth honesty (T3)
reproducible	Numbers re-derive from a checksummed corpus + public scoring code	Re-export, verify checksum, re-run scoreBacktest (the verify-this-review walkthrough)	Corpus is publishable and published
asserted	LoopOver's own statement; no independent artifact	Nothing	Full operator trust — the weakest tier, labeled so it can be discounted

Everything shipping today is reproducible. attested becomes reachable when SNP hardware lands (#8535–#8537); the software half — harness (#9211), verifier CLI (#9212), pinned image (#9214), deploy manifests (#9213) — is already merged or in review.

A validator must be able to filter by tier. A consumer that treats all three identically has defeated the point of labeling them.

---

5. Governance of the scoring rules

Decision: scoring-rule changes gate through the same prospective-backtest machinery that already gates rule changes — no privileged path for the thing being scored.

1. Versioned: commitments.scoringRuleVersion is a monotonic semantic version, independent of the npm package version. A change to what a number means bumps it, even if code is untouched.
2. Gated: the change replays against historical corpora and must clear the Pareto floor (compareBacktestScores — any regressed axis is regressed, even when the other improves) on both the visible and held-out splits (splitBacktestCorpus, seeded). This is exactly the discipline calibration-advisory.yml already enforces.
3. Announced: version bump lands in CHANGELOG.md with the old→new semantics diff, before records carrying the new version are served. Consumers are never surprised by a number changing meaning under a stable version.
4. Immutable history: old records keep their original scoringRuleVersion. Scores are never retroactively restated.

---

6. What the subnet side runs vs trusts

Minimum to consume trustlessly — three commands, none of which require trusting us:

# 1. Fetch (or accept from anyone — the record is self-describing)
curl -s "$LOOPOVER/v1/public/eval-scores?subject=$AGENT" > scores.json

# 2. Reproducible tier: re-derive the numbers yourself
npx tsx scripts/backtest-corpus-export.ts --rule-id "$RULE" --output corpus.json --remote
#   verify corpus.json's checksum matches commitments.corpusChecksum, then re-run
#   scoreBacktest from the public @loopover/engine package and compare.

# 3. Attested tier: verify the hardware claim offline
npx tsx scripts/verify-attested-run.ts --envelope envelope.json --expected expected.json
#   exit 0 = verified; 1-8 = one failure class each; 9 = usage/IO error.

Runs: the verifier CLI and the public scoring functions — both already published, both runnable with no API key and no network path to us (step 3 is fully offline).

Trusts: AMD's root certificate (for attested), and that the recorded ground-truth override history is honest (T3 — closed by no tier, stated plainly rather than papered over).

Does not need: an account, a token, our uptime, or our good behavior at verification time.

---

Follow-ups this spec surfaces (not in scope here)

• No artifact signing exists — §1/§2. If signed bundles become a requirement, that is its own issue with a key-distribution decision attached.
• No external ledger anchor exists — buildLedgerAnchorPayload is unused; the tamper-evident/tamper-proof gap in migrations/0180 stands until anchoring ships.
• /v1/public/eval-scores does not exist yet — this spec defines it; implementation is its own issue, and benchmark: frozen-repo eval harness — score any agent's proposed maintainer actions against realized history #9216 must not emit benchmark_run records before both land.