feat(bench): GEPA over the analyst/steerer prompt (canonical stack, real agent-eval primitives)

[drewstone]

What

The flywheel, on the canonical loop system. The analyst is the steerer — observe()'s findings → recommended_action → the depth steer — so this evolves the analyst's system instruction against the live EOPS gate.

• observe() is now tunable: analystInstruction? override + exported defaultAnalystInstruction. The analyst prompt is the GEPA knob. The firewall stays structural (the observe input carries no score), so a custom instruction can't break it.
• agentic.ts: AgenticOptions.analystInstruction threads into the depth steerer.
• eops-gepa.mts: assembles the GEPA loop from agent-eval's real primitives — buildReflectionPrompt + parseReflectionResponse (reflective mutation) + paretoFrontier (selection over [maximize lift, minimize cost]). No hand-rolled optimizer. (There is no turnkey runPromptEvolution in agent-eval 0.83 — only the primitives — so the population loop is thin orchestration over them.)

Fitness = the depth-vs-breadth lift on the canonical Supervisor+observe() gate. Breadth is computed once per task (shared baseline — breadth has no analyst — correct design + halves cost). The failing per-task lifts are the reflection gradient. Seeds = observe()'s proven default (the +16.4pp instruction) FIRST, then the designer-panel population — so GEPA improves from known-good, not from below baseline.

Validation

Smoke (N=2, 1 gen) ran the full loop end-to-end: score → paretoFrontier select → buildReflectionPrompt→LLM→parseReflectionResponse → child → re-score → pick. Bounded real run (N=6, 2 gens, maxShots=3, deepseek-v4-pro) in flight — will report whether GEPA finds an analyst prompt beating the seeded baseline.

Test

typecheck clean (runtime + bench, 0 errors); observe() change is additive (default preserved); smoke validated the loop.

[tangletools]

❌ Needs Work — dfca5406

Readiness 61/100 · Confidence 70/100 · 9 findings (1 high, 8 low)

	deepseek	glm	aggregate
Readiness	61	80	61
Confidence	70	70	70
Correctness	61	80	61
Security	61	80	61
Testing	61	80	61
Architecture	61	80	61

Full multi-shot audit completed 2/2 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 2/2 planned shots over 4 changed files. Global verifier still owns final merge decision.

Blocking

🔴 HIGH Missing population file makes eops-gepa.mts always crash on startup — bench/src/eops-gepa.mts

Line 116: readFileSync('steerers/eops-itsm-population.json', 'utf8') — the steerers/ directory does not exist in the repo, so this always throws ENOENT. The script never reaches the evaluation loop. The defaultAnalystInstruction seed on line 119 is the only required seed; the file-based seeds are supplementary. Either (a) check in the population file, or (b) wrap in try/catch and default to an empty array so the script can run with just the observe-default seed.

Other

🟡 LOW No test coverage for analystInstruction propagation — bench/src/agentic.ts

Line 193: The conditional spread ...(opts.analystInstruction ? { analystInstruction: opts.analystInstruction } : {}) is the critical seam that lets GEPA control the steerer, but has zero test coverage. The observe() function's analystInstruction override is also untested. Since this is bench tooling, not a library hot path, the risk is low, but a single integration test confirming the knob reaches observe() would raise confidence.

🟡 LOW Whitespace-only analystInstruction bypasses truthiness guard in agentic.ts — bench/src/agentic.ts

Line 193: the spread guard ...(opts.analystInstruction ? { analystInstruction: opts.analystInstruction } : {})) passes the ?? in coercible values (e.g. ' '), which would be sent as an effectively empty system prompt to observe(). The eops-gepa.mts caller defends against this with instruction.trim().length < 40 at line 165, but agentic.ts itself has no guard. Consider opts.analystInstruction?.trim() ? ... to match the intent.

🟡 LOW Hardcoded relative path for seed population file — bench/src/eops-gepa.mts

Line 116: readFileSync('steerers/eops-itsm-population.json', 'utf8') uses a relative path, so the script must be run from bench/ or the read fails. The file exists (bench/steerers/eops-itsm-population.json), and the usage docstring (line 17) shows running as tsx src/eops-gepa.mts from bench/, so this works in practice. Low severity — could use import.meta.dirname for robustness but not blocking.

🟡 LOW Reflection top/bottom trials overlap when perTask has ≤2 entries — bench/src/eops-gepa.mts

Lines 144-146: sorted.slice(0, 2) (top) and sorted.slice(-2) (bottom) produce identical arrays when perTask has exactly 2 items, feeding the reflection prompt no gradient signal. With N=4 tasks and infrastructure skips, this can happen. Impact: degraded optimization quality, not correctness. Fix: guard with if (sorted.length <= 2) { bottom = sorted.slice(0, 1); top = sorted.slice(-1); } or skip reflection for that parent.

🟡 LOW Custom analystInstruction can remove behavioral guardrails — src/runtime/observe.ts

The analystInstruction option replaces the full system prompt, including the constraints 'Only claim what the trace shows' and 'No findings if the run was clean' (lines 57-62). While the score firewall is structural (ObserveInput has no score field; derived_from_judge is hardcoded false at line 174), a custom instruction can instruct the analyst to hallucinate findings. This is the intended use case (GEPA optimization surface) and is honestly documented in the JSDoc ([lines 48-53](https://github.com/tangle-network/agent-runtime/blob/dfca5406

🟡 LOW No test coverage for observe() function — src/runtime/observe.ts

No test file exists for src/runtime/observe.ts (glob for observe.test.* / observe.spec.* returned no results). The observe() function — including its new analystInstruction path — has zero automated coverage. This is a pre-existing condition, not introduced by this PR, but the new code path (analystInstruction override, defaultAnalystInstruction export) inherits the gap. The bench/src/eops-gepa.mts integration provides smoke coverage but no unit-level assertion on the option plumbing.

🟡 LOW No unit test for observe() or the analystInstruction override — src/runtime/observe.ts

There is no test file for observe() anywhere in the repo (checked tests/**/*observe* and grep for imports). The new analystInstruction fallback path (opts.analystInstruction ?? defaultAnalystInstruction) is untested. The function is non-trivial (LLM call, JSON parse, corpus append). This is pre-existing tech debt, not introduced by this PR, but the new parameter adds a coverage gap. A minimal test mocking ChatClient would confirm the override flows through and the fallback fires when omitted.

🟡 LOW Verbose doc comment on analystInstruction could be trimmed — src/runtime/observe.ts

Lines 48-53: the JSDoc for analystInstruction is 6 lines explaining GEPA context and the firewall invariant. The same rationale is repeated in the module-level doc (lines 1-17) and in the defaultAnalystInstruction export comment. Not a bug, but the redundancy adds maintenance surface. Consider a one-liner: /** Override the analyst system instruction. Omitted ⇒ default. */ — the firewall explanation belongs in module docs, not per-field.

---

_{tangletools · 2026-06-09T11:43:33Z · trace}

[tangletools]

❌ 1 Blocking Finding — dfca5406

Full multi-shot audit completed 2/2 planned shots over 4 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 2/2 planned shots over 4 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-06-09T11:43:33Z · immutable trace}

[tangletools]

Premise check withheld merge — dfca5406

Classifier flagged this PR as a premise claim (numeric pp/% delta + eval terminology). Confidence: medium.

Recommend re-running the underlying eval with pairedEvalueSequence before merging.

• Cited claim: +16.4pp
• PR body excerpt: feat(bench): GEPA over the analyst/steerer prompt (canonical stack, real agent-eval primitives)

Run:

pnpm eval:evolve --reps 5 --skip-mutation

Classifier rationale: Body cites 1 numeric claim(s) (+16.4pp) and eval-related terms appear in pr_body, review_findings. PR is asserting a measurable result that repair-pr cannot polish away — re-run the underlying evaluation before merging.

---

_{tangletools premise check · #205}

[tangletools]

✅ No Blockers — 1dfbfd67

Readiness 67/100 · Confidence 75/100 · 12 findings (2 medium, 10 low)

	deepseek	glm	aggregate
Readiness	73	67	67
Confidence	75	75	75
Correctness	73	67	67
Security	73	67	67
Testing	73	67	67
Architecture	73	67	67

Full multi-shot audit completed 3/3 planned shots over 6 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 6 changed files. Global verifier still owns final merge decision.

🟠 MEDIUM adaptiveRefine double-opens surface on first-shot score=0 when best=-1 initial condition causes premature abandon — bench/src/agentic.ts

Line 510: let best = -1. If the very first shot scores 0 (passes=0), then line 519 out.score <= best is 0 <= -1 → false, so best becomes 0. But if the SECOND shot on a fresh restart also scores 0, 0 <= 0 is true → abandons and restarts again. This means the strategy can thrash: open→score 0→continue→open→score 0→abandon→restart→repeat. The initial best = -1 sentinel means the first 0-score shot always passes the check, but subsequent 0-score shots on the same handle trigger restart. This is probably the intended 'branch-when-stuck' behavi

🟠 MEDIUM runBenchmark crashes when all tasks are excluded via transient infra — bench/src/run-benchmark.mts

Line 84: pairedLift(ok.map(...), ok.map(...)) throws pairedLift: no pairs (stats.mts:69) when ok is empty (all tasks excluded). The function's doc claims 'Resilient: a task whose rollouts fail is excluded, not fatal' but if ALL fail, it crashes with an unhandled exception. Reproduction: gym container down, network outage, or all tasks hitting router auth failures. Fix: guard the pairedLift call with if (ok.length > 0) or catch the error and set refineVsSample = undefined.

🟡 LOW AgenticRunResult.mode widened from union to string — no exhaustiveness check — bench/src/agentic.ts

Line 312: mode: string (was 'depth' | 'breadth'). This is intentional to support custom strategy names, but any consumer doing switch(result.mode) with only 'depth'/'breadth' cases won't get a TS exhaustiveness error for the new 'adaptiveRefine' or custom names. This is the expected tradeoff for the Strategy extension point, but downstream code that pattern-matches on mode should be audited (outside this shot's scope).

🟡 LOW No tests for new Strategy/defineStrategy/adaptiveRefine/runBenchmark abstractions — bench/src/agentic.ts

None of the 11 test files in bench/src/ import or exercise the new public API surface: Strategy, defineStrategy, adaptiveRefine, sample, refine as Strategy objects, runBenchmark, printBenchmarkReport. The existing refine-loop.test.mts tests a different abstraction (runRefineLoop). The Strategy abstraction is now the primary extension point and should have at minimum a test verifying defineStrategy produces a working driver, and that adaptiveRefine correctly handles the restart-then-resume flow.

🟡 LOW adaptiveRefine restart never resets best-score threshold, causing potential restart spiral — bench/src/agentic.ts

Lines 519-525: When a steered shot scores <= best and the line is abandoned/restarted, best retains the previous line's best score. The next line's first shot (on a fresh artifact with minimal work done) is compared against this inherited best, and if it falls short — which is likely since the artifact starts from scratch — the line is immediately abandoned again. This conflates two separate concerns into one variable: global-best-across-lines and per-line-improvement-threshold. The restart loop continues without any line getting more than one shot, wasting budget. Fix: track globalBest separately from a per-line improvement threshold, or reset `best = out

🟡 LOW eops-gepa.mts assumes cwd=bench/ for seed file path — bench/src/eops-gepa.mts

Line 116: readFileSync('steerers/eops-itsm-population.json') is a relative path resolved against process.cwd(). The documented invocation (tsx src/eops-gepa.mts) implies running from the bench/ directory, but if run from the repo root or any other directory, the file won't be found and the process exits with a raw ENOENT error. Fix: use import.meta.url-based resolution or a CLI flag for the population file path.

🟡 LOW eops-gepa.mts fitness silently returns -1 lift when ALL tasks fail depth — bench/src/eops-gepa.mts

Line 111: lift: scored ? liftSum / scored : -1 — if every task's depth run throws, scored=0 and the candidate gets lift=-1, cost=1e9. This is a safe sentinel that won't win the pareto frontier, but the error message at line 91 only guards breadth baseline (< 2 tasks). If all depth runs fail for every candidate, GEPA runs to completion with all candidates at lift=-1 and the 'winner' is arbitrary. No data loss, but the user gets a misleading 'WINNER' log. A warning when scored === 0 would help.

🟡 LOW eops-gepa.mts reads seed population from hardcoded relative path — bench/src/eops-gepa.mts

Line 116: readFileSync('steerers/eops-itsm-population.json', ...) — this is a relative path resolved from CWD, not from import.meta.dirname. The file exists at bench/steerers/, so it only works if CWD is bench/. The rest of the file uses absolute env vars (EOPS_GYM_DBS_DIR). If someone runs tsx bench/src/eops-gepa.mts from the repo root, this throws ENOENT. Should use path.resolve(import.meta.dirname, '../steerers/eops-itsm-population.json') or make the path an env var.

🟡 LOW runBenchmark sequentially runs strategies per task — no isolation guarantee for custom strategies — bench/src/run-benchmark.mts

Lines 65-69: for (const s of strategies) { const r = await runAgentic({...}) } — strategies run sequentially per task. Built-in strategies (sample/refine) open fresh artifacts, so there's no shared-state contamination. But a custom Strategy could mutate global or surface-level state (e.g., writing files) that bleeds into the next strategy's run on the same task. The benchmark doesn't document this sequencing assumption. Worth a note in BenchmarkConfig.strategies JSDoc that strategies must not share mutable state.

🟡 LOW Empty-string analystInstruction bypasses default silently — src/runtime/observe.ts

Line 148: opts.analystInstruction ?? defaultAnalystInstruction uses nullish coalescing, which passes through "" as-is, resulting in an empty system prompt. This would produce degenerate LLM output. The only current caller (bench/src/agentic.ts:193) guards with opts.analystInstruction ? { analystInstruction: ... } : {}, so it's safe in practice, but a future caller could pass "" accidentally. Fix: either validate non-empty in observe() or switch to || (which also catches ""). This is a nit, not a blocker.

🟡 LOW No test coverage for observe() or the new analystInstruction option — src/runtime/observe.ts

No test files exist for observe.ts (globbed *observe*.test.* and *observe*.spec.* — zero results). The new analystInstruction option and the defaultAnalystInstruction export are untested. This is pre-existing (no tests were removed), but the public API surface grows without coverage. A unit test that stubs ChatClient and verifies the system message content with/without the override would be low-effort and high-value.

🟡 LOW No test coverage for exported defaultAnalystInstruction — src/runtime/observe.ts

The new defaultAnalystInstruction constant is exported and consumed by bench/src/eops-gepa.mts (line 27) as a seed for GEPA prompt optimization, but no test asserts its content or verifies the ?? fallback behavior in observe(). Pre-existing gap (no observe.test.ts existed before this PR), but the export raises the blast radius if the string is ever accidentally modified.

---

_{tangletools · 2026-06-09T12:13:24Z · trace}

Superseded by re-review — no blocking findings on latest commit.

[tangletools]

✅ Approved — 12 non-blocking findings — 1dfbfd67

Full multi-shot audit completed 3/3 planned shots over 6 changed files. Global verifier still owns final merge decision. | Full multi-shot audit completed 3/3 planned shots over 6 changed files. Global verifier still owns final merge decision.

Full immutable report for this review: trace

Summary comment for this run: full summary

---

_{tangletools · 2026-06-09T12:13:24Z · immutable trace}

[tangletools]

Premise check withheld merge — 1dfbfd67

Classifier flagged this PR as a premise claim (numeric pp/% delta + eval terminology). Confidence: medium.

Recommend re-running the underlying eval with pairedEvalueSequence before merging.

• Cited claim: +16.4pp
• PR body excerpt: feat(bench): GEPA over the analyst/steerer prompt (canonical stack, real agent-eval primitives)

Run:

pnpm eval:evolve --reps 5 --skip-mutation

Classifier rationale: Body cites 1 numeric claim(s) (+16.4pp) and eval-related terms appear in pr_body, review_findings. PR is asserting a measurable result that repair-pr cannot polish away — re-run the underlying evaluation before merging.

---

_{tangletools premise check · #205}

[tangletools]

✅ Refreshed approval after new commits — ab137984

A previous trusted approval on this PR was invalidated by new commits.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: stale_approval_refresh · 2026-06-09T12:16:09Z}

[tangletools]

⚠️ Review Interrupted — ab137984

The review runner stopped before publishing a final verdict: webhook_restarted.

State	Detail
Interrupted	webhook restarted

No review verdict was produced for this run. Trigger a fresh review on the current PR head if the PR is still open.

_{tangletools · #205 · model: kimi-for-coding · updated 2026-06-09T12:30:05Z}

[tangletools]

✅ Auto-approved PR — a7e18a15

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-09T12:40:24Z}

[tangletools]

✅ Auto-approved PR — db0e0821

Blanket team auto-approval is enabled for this reviewer service.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: blanket_auto_approve · 2026-06-09T12:42:00Z}