Skip to content

test: live-backend harness for the contract suite (core + multisig)#673

Merged
0xisk merged 58 commits into
mainfrom
test/multisig-live-harness
Jul 18, 2026
Merged

test: live-backend harness for the contract suite (core + multisig)#673
0xisk merged 58 commits into
mainfrom
test/multisig-live-harness

Conversation

@0xisk

@0xisk 0xisk commented Jul 10, 2026

Copy link
Copy Markdown
Member

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Closes #666
Closes #667

Stands up a live-backend test harness so the existing unit specs can run
against a local Midnight node (make env-up, MIDNIGHT_BACKEND=live, the
unit-live vitest project), and brings the multisig shielded specs green on it.
First two sub-issues of the umbrella #665.

What's here

  • Live backend + harness (contracts/test-utils/harness/): WalletPool,
    FundedWallet, LiveSimulatorBackend (deploy-per-create(), per-alias
    providers), plus funding / dust / network / ledgerEvents /
    NativeShieldedTokenTracker and the live.setup.ts composition root.
  • test-utils reorg: fixtures/ (address, zswap, shieldedKey,
    nativeShieldedToken) split from harness/; vitest configs consolidated into
    one vitest.config.ts with unit / unit-live / integration / harness
    projects.
  • Preset rename by capability: ShieldedMultiSigShieldedProposalMultisig,
    ShieldedMultiSigV2ShieldedStatelessMultisig,
    ShieldedMultiSigV3NativeShieldedTokenMultisig (CHANGELOG updated).
  • Fixes found standing it up:
    • share one in-memory private-state store across signer aliases (was
      per-alias LevelDB lock contention + coin-PK scoping);
    • fund the 4th signer the dev preset leaves empty (top-up from the deployer);
    • re-sync the wallet before balancing a tx, fixing a stale-UTXO race that
      surfaced as a bare "Transaction submission error" on consecutive
      same-signer submissions.
  • Raise the live testTimeout to 600s for the ~10-tx lifecycle specs.

Scope note

Large diff, but most of it is mechanical: test-utils import-path updates across
suites, yarn.lock, and the vitest config merge. The substantive new code is
test-utils/harness/ + test-utils/fixtures/. No production contract behavior
changes beyond the preset renames.

PR Checklist

  • I have read the Contributing Guide
  • I have added tests that prove my fix is effective or that my feature works.
  • I have added documentation for new methods or changes to existing method behavior.
  • CI Workflows Are Passing

Further comments

Opened as a draft: dry suite, tsc, and biome are green, and the heaviest
live subset (full lifecycle, 3 tests) passes with the sync fix. Flipping to
ready once the full ShieldedProposalMultisig live suite (and the rest of the
multisig live specs) are confirmed green on a fresh node. The remaining category
sub-issues (#668#672) roll out one at a time under #665.

Summary by CodeRabbit

  • New Features

    • Added native shielded-token fixtures and support for dry and live test environments.
    • Added live test tooling with wallet funding, progress reporting, retry handling, and environment safeguards.
    • Added expanded multisig coverage for proposal-based and stateless variants.
  • Updates

    • Replaced Compact workflows with Compile workflows across build and test commands.
    • Updated multisig naming and simplified proposal and treasury access patterns.
  • Documentation

    • Added comprehensive test-running guidance and refreshed multisig changelog details.
  • Bug Fixes

    • Corrected bug-report instructions and disabled an unreliable proof-server health check.

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 28e82e8c-f1a0-4b5e-b18e-272ed776538f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Walkthrough

This PR migrates repository workflows from compact to compile, adds a live-backend test harness, introduces backend-aware fixtures, and updates multisig contracts, simulators, and tests for native, proposal, and stateless presets.

Changes

Repository build and test workflows

Layer / File(s) Summary
Compile, lint, and test command migration
.github/workflows/*, package.json, contracts/package.json, turbo.json, contracts/vitest.config.ts
CI and workspace scripts now use compile and check commands, with explicit Vitest projects for unit, integration, harness, and live tests.
Testing documentation and environment setup
CONTRIBUTING.md, local-env.yml
Test-running instructions now cover unit/live workflows, and the proof-server healthcheck is disabled.

Multisig contracts, simulators, and tests

Layer / File(s) Summary
Preset and simulator migration
contracts/src/multisig/presets/*, contracts/src/multisig/test/simulators/*
Multisig naming and simulator wiring move to native, proposal, and stateless contract artifacts using empty private state and witnesses.
Backend-aware multisig coverage
contracts/src/multisig/test/*, contracts/src/token/test/*
Tests use native token fixtures, qualified shielded coins, fresh simulator instances, and separate dry/live assertions.
Shared test fixtures
contracts/test-utils/fixtures/*
Address, shielded-key, native-token, and Zswap helpers and their tests are added.

Live backend harness

Layer / File(s) Summary
Wallet and backend infrastructure
contracts/test-utils/harness/FundedWallet.ts, WalletPool.ts, LiveSimulatorBackend.ts, funding.ts, dust.ts
The harness builds funded wallets, manages aliases and dust, deploys compiled artifacts, routes providers, and retries transient deployment failures.
Indexer and live execution support
contracts/test-utils/harness/*, scripts/test-live.ts, scripts/keyIntegrity.ts
Indexer event utilities qualify shielded coins, live setup enforces fresh-node and lock conditions, and live tests include artifact repair and flaky-failure reruns.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

Possibly related PRs

Suggested labels: enhancement

Suggested reviewers: emnul, andrew-fleming, pepebndc, dependabot[bot]

Sequence Diagram(s)

sequenceDiagram
  participant LiveSetup
  participant WalletPool
  participant LiveSimulatorBackend
  participant LocalNode
  LiveSetup->>WalletPool: build and fund worker wallets
  WalletPool->>LiveSimulatorBackend: register wallet aliases
  LiveSimulatorBackend->>LocalNode: compile and deploy artifact
  LocalNode-->>LiveSimulatorBackend: deployed contract address
  LiveSimulatorBackend-->>LiveSetup: live simulator context
Loading

Poem

A rabbit checks the compile trail,
While wallets hop through live-test mail.
Coins find keys, and proofs take flight,
Multisigs bloom in lantern light.
“Hop onward!” says the bunny bright.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.76% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: enabling live-backend harness support for core and multisig contract tests.
Linked Issues check ✅ Passed The PR delivers the live harness, wallet/funding helpers, shared state fixes, multisig live specs, and 600s timeouts required by #666 and #667.
Out of Scope Changes check ✅ Passed The changed docs, CI, and test scripts support the live harness rollout and multisig live execution rather than introducing unrelated features.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/multisig-live-harness

Comment @coderabbitai help to get the list of available commands.

@0xisk
0xisk marked this pull request as ready for review July 13, 2026 16:57
@0xisk
0xisk requested review from a team as code owners July 13, 2026 16:57
@0xisk 0xisk self-assigned this Jul 13, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (2)
contracts/vitest.config.ts (2)

43-52: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate MAX_LIVE_WORKERS constant.

WalletPool.ts already exports MAX_LIVE_WORKERS = 3 (used by live.setup.ts to bound VITEST_POOL_ID). Redefining the same cap here risks the two drifting apart silently.

♻️ Import the shared constant instead of redefining it
-const MAX_LIVE_WORKERS = 3;
+import { MAX_LIVE_WORKERS } from './test-utils/harness/WalletPool.js';
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@contracts/vitest.config.ts` around lines 43 - 52, Remove the local
MAX_LIVE_WORKERS declaration in vitest.config.ts and import the shared exported
MAX_LIVE_WORKERS from WalletPool.ts. Keep the existing liveWorkers calculation
unchanged so it continues using the shared cap.

135-146: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

harness-live lacks the groupOrder isolation given to unit-live.

Per the comment on unit-live (lines 113-116), Vitest rejects a multi-project run where two projects share a group but differ in maxWorkers. harness-live's fileParallelism: false implicitly forces maxWorkers: 1, differing from unit/integration/harness's defaults in the same (unset) group — the same hazard unit-live was given groupOrder: 1 to avoid. Only a problem if all projects are ever run together without --project, but worth closing for consistency.

♻️ Give harness-live its own group
         test: {
           ...NODE,
           ...LIVE_SEQUENTIAL,
           name: 'harness-live',
           include: ['test-utils/**/*.test.ts'],
           globalSetup: ['./test-utils/harness/live.globalSetup.ts'],
+          sequence: { ...LIVE_SEQUENTIAL.sequence, groupOrder: 2 },
         },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@contracts/vitest.config.ts` around lines 135 - 146, Add the same isolated
groupOrder configuration used by unit-live to the harness-live project entry.
Keep its existing LIVE_SEQUENTIAL, project name, include pattern, and
globalSetup unchanged, ensuring harness-live is separated from harness and other
projects with different worker settings.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@contracts/src/multisig/test/ShieldedTreasury.test.ts`:
- Around line 30-33: Correct the comment above Z_RECIPIENT to state that it is
assigned in beforeAll, matching the initialization at beforeAll and the nearby
“resolve it once after the first deploy” comment. Preserve the existing
explanation of live wallet synchronization and dry-run behavior.

In `@contracts/src/multisig/test/simulators/ShieldedProposalMultisigSimulator.ts`:
- Around line 142-149: Update getReceivedMinusSent to await
getReceivedTotal(color) before invoking getSentTotal(color), ensuring the two
impure simulator queries run sequentially; preserve the existing subtraction and
returned bigint result.

In `@contracts/test-utils/harness/FundedWallet.ts`:
- Around line 92-116: Wrap provider startup, funding, synchronization, and
balanceTx setup in a try/catch within the FundedWallet initialization flow, and
call provider.stop() when any step fails before rethrowing the original error.
Ensure cleanup also covers failures from start, waitForFunds, and syncWallet,
while preserving the successful return of the FundedWallet instance.

In `@contracts/test-utils/harness/network.ts`:
- Around line 9-13: Replace the direct Number conversions in PORTS with shared
validation that accepts only integer port values from 1 through 65535, rejecting
empty, malformed, fractional, zero, and out-of-range overrides. When validation
fails, throw an error identifying the responsible environment-variable name,
while preserving the existing defaults for unset variables.

In `@contracts/test-utils/harness/test/network.test.ts`:
- Around line 11-18: Update the “should default to the local stack ports” test
in the PORTS describe block to isolate it from environment overrides: save and
clear all supported MIDNIGHT_*_PORT variables before dynamically importing
network.js, assert the imported PORTS defaults, and restore the original
environment values afterward even if the assertion fails.

In `@contracts/test-utils/harness/WalletPool.ts`:
- Around line 104-114: Update WalletPool.ensureReady to build wallets
sequentially instead of using Promise.all over Object.entries(this.seeds).
Preserve the existing per-wallet setup—await buildWallet, set coinPkEnv(alias),
and store the wallet in this.wallets—while ensuring each builder completes
before starting the next so shared-deployer funding cannot race.

In `@turbo.json`:
- Around line 60-69: Update the compile task’s dependsOn list in turbo.json to
include compile:archive, preserving the existing compile dependencies so
aggregate compile and its dependent test tasks rebuild archive artifacts.

---

Nitpick comments:
In `@contracts/vitest.config.ts`:
- Around line 43-52: Remove the local MAX_LIVE_WORKERS declaration in
vitest.config.ts and import the shared exported MAX_LIVE_WORKERS from
WalletPool.ts. Keep the existing liveWorkers calculation unchanged so it
continues using the shared cap.
- Around line 135-146: Add the same isolated groupOrder configuration used by
unit-live to the harness-live project entry. Keep its existing LIVE_SEQUENTIAL,
project name, include pattern, and globalSetup unchanged, ensuring harness-live
is separated from harness and other projects with different worker settings.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9c3d96c3-a237-41e7-957c-97139d2cf02b

📥 Commits

Reviewing files that changed from the base of the PR and between 9943e35 and 1b8f51a.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (84)
  • .github/ISSUE_TEMPLATE/01_bug_report.yml
  • .github/workflows/checks.yml
  • .github/workflows/test.yml
  • CHANGELOG.md
  • CONTRIBUTING.md
  • contracts/package.json
  • contracts/src/access/test/AccessControl.test.ts
  • contracts/src/access/test/Ownable.test.ts
  • contracts/src/access/test/ZOwnablePK.test.ts
  • contracts/src/multisig/presets/NativeShieldedTokenMultisig.compact
  • contracts/src/multisig/presets/ShieldedProposalMultisig.compact
  • contracts/src/multisig/presets/ShieldedStatelessMultisig.compact
  • contracts/src/multisig/test/EmptyWitnesses.ts
  • contracts/src/multisig/test/Forwarder.test.ts
  • contracts/src/multisig/test/ForwarderPrivate.test.ts
  • contracts/src/multisig/test/NativeShieldedTokenMultisig.test.ts
  • contracts/src/multisig/test/ProposalManager.test.ts
  • contracts/src/multisig/test/ShieldedMultiSig.test.ts
  • contracts/src/multisig/test/ShieldedProposalMultisig.test.ts
  • contracts/src/multisig/test/ShieldedStatelessMultisig.test.ts
  • contracts/src/multisig/test/ShieldedTreasury.test.ts
  • contracts/src/multisig/test/ShieldedTreasuryStateless.test.ts
  • contracts/src/multisig/test/Signer.test.ts
  • contracts/src/multisig/test/SignerManager.test.ts
  • contracts/src/multisig/test/presets/ForwarderPrivate.test.ts
  • contracts/src/multisig/test/presets/ForwarderShielded.test.ts
  • contracts/src/multisig/test/presets/ForwarderUnshielded.test.ts
  • contracts/src/multisig/test/simulators/MockShieldedTreasuryStatelessSimulator.ts
  • contracts/src/multisig/test/simulators/NativeShieldedTokenMultisigSimulator.ts
  • contracts/src/multisig/test/simulators/ProposalManagerSimulator.ts
  • contracts/src/multisig/test/simulators/ShieldedProposalMultisigSimulator.ts
  • contracts/src/multisig/test/simulators/ShieldedStatelessMultisigSimulator.ts
  • contracts/src/multisig/test/simulators/ShieldedTreasurySimulator.ts
  • contracts/src/multisig/test/simulators/SignerManagerSimulator.ts
  • contracts/src/multisig/test/simulators/SignerSimulator.ts
  • contracts/src/multisig/test/witnesses/ProposalManagerWitnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigV2Witnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigV3Witnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigWitnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedTreasuryWitnesses.ts
  • contracts/src/multisig/test/witnesses/SignerManagerWitnesses.ts
  • contracts/src/multisig/test/witnesses/SignerWitnesses.ts
  • contracts/src/multisig/test/witnesses/UnshieldedTreasuryWitnesses.ts
  • contracts/src/token/test/FungibleToken.test.ts
  • contracts/src/token/test/MultiToken.test.ts
  • contracts/src/token/test/NativeShieldedToken.test.ts
  • contracts/src/token/test/NativeShieldedTokenCore.test.ts
  • contracts/src/token/test/NativeShieldedTokenFamily.test.ts
  • contracts/src/token/test/nonFungibleToken.test.ts
  • contracts/src/utils/test/utils.test.ts
  • contracts/test-utils/fixtures/address.ts
  • contracts/test-utils/fixtures/nativeShieldedToken.ts
  • contracts/test-utils/fixtures/shieldedKey.ts
  • contracts/test-utils/fixtures/test/address.test.ts
  • contracts/test-utils/fixtures/test/nativeShieldedToken.test.ts
  • contracts/test-utils/fixtures/test/shieldedKey.test.ts
  • contracts/test-utils/fixtures/test/zswap.test.ts
  • contracts/test-utils/fixtures/zswap.ts
  • contracts/test-utils/harness/FundedWallet.ts
  • contracts/test-utils/harness/LiveSimulatorBackend.ts
  • contracts/test-utils/harness/NativeShieldedTokenTracker.ts
  • contracts/test-utils/harness/WalletPool.ts
  • contracts/test-utils/harness/dust.ts
  • contracts/test-utils/harness/funding.ts
  • contracts/test-utils/harness/ledgerEvents.ts
  • contracts/test-utils/harness/live.globalSetup.ts
  • contracts/test-utils/harness/live.setup.ts
  • contracts/test-utils/harness/liveProgressReporter.ts
  • contracts/test-utils/harness/network.ts
  • contracts/test-utils/harness/test/FundedWallet.test.ts
  • contracts/test-utils/harness/test/LiveSimulatorBackend.test.ts
  • contracts/test-utils/harness/test/NativeShieldedTokenTracker.test.ts
  • contracts/test-utils/harness/test/WalletPool.test.ts
  • contracts/test-utils/harness/test/dust.test.ts
  • contracts/test-utils/harness/test/liveGlobalSetup.test.ts
  • contracts/test-utils/harness/test/network.test.ts
  • contracts/vitest.config.ts
  • contracts/vitest.integration.config.ts
  • contracts/vitest.live.config.ts
  • local-env.yml
  • package.json
  • scripts/keyIntegrity.ts
  • scripts/test-live.ts
  • turbo.json
💤 Files with no reviewable changes (11)
  • contracts/vitest.live.config.ts
  • contracts/src/multisig/test/witnesses/SignerManagerWitnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigV2Witnesses.ts
  • contracts/src/multisig/test/witnesses/SignerWitnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedTreasuryWitnesses.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigV3Witnesses.ts
  • contracts/src/multisig/test/witnesses/ProposalManagerWitnesses.ts
  • contracts/src/multisig/test/witnesses/UnshieldedTreasuryWitnesses.ts
  • contracts/vitest.integration.config.ts
  • contracts/src/multisig/test/witnesses/ShieldedMultiSigWitnesses.ts
  • contracts/src/multisig/test/ShieldedMultiSig.test.ts

Comment thread contracts/src/multisig/test/ShieldedTreasury.test.ts Outdated
Comment thread contracts/src/multisig/test/simulators/ShieldedMultiSigSimulator.ts Outdated
Comment thread contracts/test-utils/harness/FundedWallet.ts Outdated
Comment thread contracts/test-utils/harness/network.ts
Comment thread contracts/test-utils/harness/test/network.test.ts
Comment thread contracts/test-utils/harness/WalletPool.ts
Comment thread turbo.json
@0xisk
0xisk force-pushed the test/multisig-live-harness branch from 62fc4e8 to 0ae9c1c Compare July 14, 2026 13:28
@0xisk

0xisk commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

@andrew-fleming added docs on how to run live test per specific test file. 2bdab60

@0xisk

0xisk commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

CI for the live suite: live.yml

The last 9 commits add a CI workflow for the live tests. It is separate from the normal checks and never required, since a full category takes hours, not minutes.

Triggers

  • Nightly on main (02:00 UTC). A failed nightly opens or comments on a live-nightly tracking issue, which closes automatically on the next green run.
  • On demand: workflow_dispatch with a category dropdown (all / multisig) and an optional vitest file filter.
  • On a PR: apply the live-tests label (re-apply to re-run). Since only maintainers can label, this also gates fork code.

Shape

  • A small plan job derives the live-ready list from the runner itself (yarn test:live --list, backed by LIVE_READY in scripts/test-live.ts), so there is no second list to maintain.
  • A matrix fans out one parallel job per category (live-multisig, ...). Each gets its own runner, docker stack, compile, and 6-hour job budget; fail-fast: false, so one category cannot cancel the rest. Wall clock is the slowest category, not the sum.
  • Each job runs the same yarn test:live runner as a local run, scoped to its category (yarn test:live <category>, the matrix value as the first arg): compile with ZK-key verification, stack reset via make env-up, harness smoke, then the two-round flake classification (flaky = exit 0, reported loudly). Only a bare local yarn test:live runs every category in one invocation; in CI the all-categories view is the sum of the parallel jobs.

Observability

  • Each job writes its verdict (with flaky/real file lists) to the GitHub job summary, and every flaky file gets a ::warning:: annotation, so a green-but-flaky run stays visible.
  • Artifacts: live-reports-<category> (JSON verdicts, uploaded on every run, 14 days) and live-logs-<category> (worker + service logs, failure-only, 7 days).

Update (1bc30b3, 6980a49)

  • Dropped the unused test:live turbo task and the contracts test:live script: nothing invoked them since the unified runner, and they were a second entry point that skipped the stack reset, lock, and flake verdict. The runner is the single entry point for live tests.
  • The test-live.ts header now documents why the runner is a script rather than turbo tasks: turbo models a DAG of stateless, cacheable tasks, while a live run needs stateful orchestration (two-round flake verdict, docker lifecycle against one shared node, ZK-key self-heal, infra-vs-test exit codes). Turbo still handles the DAG-shaped steps inside the run (cached compile, harness smoke).

Notes

  • schedule and workflow_dispatch only activate once this lands on main; the live-tests label already works on this PR.
  • The dispatch dropdown mirrors LIVE_READY; a stale entry fails loudly (runner exit 2). New categories join the nightly and label runs automatically via --list.
  • Sizing: one file (ShieldedProposalMultisig) took ~63 min locally, so a category is a multi-hour job. If a category ever nears the 6-hour cap, the levers are sharding within the category or a larger runner.

@andrew-fleming andrew-fleming left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work, @0xisk! This is a partial review, but I left some comments and questions

Also, make env-up is not portable across Docker Compose versions (I can't highlight the makefile here). yarn test:live aborts during startup on my machine. My quick fix to get it working:

docker compose -f $(COMPOSE_FILE) up -d
docker compose -f $(COMPOSE_FILE) up -d --wait node indexer

Comment thread .github/ISSUE_TEMPLATE/01_bug_report.yml
Comment thread .github/workflows/checks.yml Outdated
Comment thread .github/workflows/live.yml Outdated
Comment thread contracts/src/multisig/presets/NativeShieldedTokenMultisig.compact Outdated
Comment thread contracts/src/multisig/test/ShieldedMultiSigV3.test.ts Outdated
Comment thread CONTRIBUTING.md Outdated
Comment thread CHANGELOG.md Outdated
Comment thread contracts/src/multisig/test/ShieldedTreasuryStateless.test.ts Outdated
Comment on lines +98 to +100
describe.skipIf(isLiveBackend())(
'_send — change coin is spendable, via Zswap I/O (dry only, no double spend)',
() => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followup: Why not just insert dry as a description in the describe block with isLiveBackend and have the dry-only tests inside of it blocks?

@0xisk 0xisk Jul 16, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate more? Because I this is already in place here: ShieldedTreasuryStateless.test.ts uses describe.skipIf(isLiveBackend())('… (dry only)') paired with describe.runIf(isLiveBackend())('… on live'), with the dry/live split right in the describe labels.

Comment thread scripts/test-live.ts
0xisk added a commit that referenced this pull request Jul 15, 2026
The live CI workflow (`live.yml`) and its "Live tests in CI"
CONTRIBUTING subsection are too large and complex to review alongside
the harness. Per review, remove them here and land them in a separate
PR stacked on this branch.

Refs: #673
0xisk added a commit that referenced this pull request Jul 15, 2026
Companion to the previous commit: the "Live tests in CI" subsection
documented and linked `live.yml`, which moves to the follow-up PR.

Refs: #673
@0xisk 0xisk mentioned this pull request Jul 15, 2026
8 tasks
0xisk added a commit that referenced this pull request Jul 15, 2026
Compiling the archive module as part of the full `compact` task fails
locally (per review). Archive is excluded from the full compile on
`main` too. Keep the `compact:archive` task for targeted runs, but drop
it from the aggregate `dependsOn`.

Refs: #673
0xisk added a commit that referenced this pull request Jul 15, 2026
`docker compose up -d --wait` waits on every service, including
proof-server, which has no healthcheck. That is not portable across
Docker Compose versions and aborts `yarn test:live` on some setups
(reported in review). Start all services, then wait only on node and
indexer, which expose healthchecks.

Refs: #673
@0xisk

0xisk commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

Fixed the make env-up portability issue in 8cce705. Root cause: proof-server has no healthcheck, so the blanket docker compose up -d --wait is not portable across Compose versions. It now starts all services, then waits only on node and indexer (the two with healthchecks), matching your suggested fix.

0xisk added a commit that referenced this pull request Jul 15, 2026
The shielded preset rename (`ShieldedMultiSig*` to capability-based
names) is a user-facing contract change unrelated to the test harness.
Per review, revert it here so this PR keeps the presets under their
existing names; the rename lands as its own stacked PR with a changelog
entry.

Harness-only content changes are preserved (e.g. the block-limit
circuit trim in ShieldedMultiSig.compact).

Refs: #673
0xisk added a commit that referenced this pull request Jul 15, 2026
Per review, enable Corepack once as a prerequisite instead of prefixing
every command with `corepack`. CI already runs `corepack enable`, so
plain `yarn` resolves to the version pinned in package.json.

* CONTRIBUTING: document the one-time `corepack enable`; drop the prefix
  from every command
* scripts/test-live.ts, keyIntegrity.ts, live.globalSetup.ts: invoke and
  print plain `yarn`

Refs: #673
0xisk added a commit that referenced this pull request Jul 15, 2026
The live runner already scopes to a category via a positional arg
(`test:live <category> [filter]`), and CI invokes it that way
(`yarn test:live "$CATEGORY"`). Drop the per-category
`test:live:multisig` script and document the argument form instead, so
local usage matches CI and the script list does not grow one entry per
category.

Refs: #673
0xisk added a commit that referenced this pull request Jul 15, 2026
Round 2 reset the shared node once, then re-ran all failed files with a
single worker. That removes parallel collisions but not sequential state
pollution: an earlier round-2 file can dirty the node (e.g. leave a spent
coin -> "Custom error: 103") and fail a later one, misclassifying a flake
as REAL. Reset the node before each failed file so every one runs alone
on a fresh node, matching the "passes in isolation" intent. Round 2 only
re-runs the failed files, so the extra resets are cheap.

Refs: #673

@andrew-fleming andrew-fleming left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, @0xisk! Thank you for moving out the unrelated changes. I left some comments and questions

Comment thread contracts/src/multisig/presets/ShieldedMultiSig.compact
Comment thread contracts/src/multisig/test/simulators/ShieldedMultiSigSimulator.ts Outdated
Comment thread contracts/src/multisig/test/simulators/ShieldedMultiSigSimulator.ts Outdated
Comment thread contracts/src/multisig/test/ShieldedTreasury.test.ts
Comment thread contracts/test-utils/harness/NativeShieldedTokenTracker.ts
Comment thread contracts/test-utils/harness/test/WalletPool.test.ts Outdated
Comment thread contracts/package.json Outdated
Comment thread CONTRIBUTING.md Outdated
Comment thread Makefile
Comment thread turbo.json
0xisk added a commit that referenced this pull request Jul 16, 2026
Per review, wrap the happy-path burn tests (real spends, dry-only for
now) in a single `describe.skipIf(isLiveBackend())('happy path (dry
only)')` instead of tagging each with `it.skipIf`, matching the
describe-based split already used in ShieldedTreasuryStateless. The
backend-agnostic reject tests stay as direct children; mint keeps its
lone skipIf (a single test needs no wrapper).

Refs: #673
0xisk added a commit that referenced this pull request Jul 16, 2026
The preset dropped getProposalRecipient/Amount/Color and getReceivedMinusSent
to fit the deploy block limit. The preset simulator re-added them as methods
that read like circuit calls but only derived the values, which is misleading.
Remove them and read the fields directly off getProposal and the received/sent
totals the preset does expose.

Refs: #673
0xisk added a commit that referenced this pull request Jul 16, 2026
Extend the tracker tests beyond the dry surface: resolve a wallet- and a
contract-owned coin to its mt_index, the anchor-block re-read, the not-found
failure, and the process-wide singleton. Only the indexer event stream is
mocked; the commitment math runs for real, so a resolve proves the tracker
computes the commitment the indexer would emit.

Refs: #673
0xisk added a commit that referenced this pull request Jul 16, 2026
The port() guard's promise (an empty, non-numeric, or out-of-range override
throws at load time, naming the env var) had no test. Add the three cases.

Refs: #673
0xisk added a commit that referenced this pull request Jul 16, 2026
sort() hid the build order, so the test could not tell serial from concurrent
building. Assert the exact order and a max concurrency of one via a counting
builder. Also fix the class doc that wrongly said "concurrently"; the builder
is serial on purpose (it funds empty signers from the shared deployer).

Refs: #673
0xisk and others added 22 commits July 17, 2026 18:00
Two operational gaps in the same workflow, batched because the hunks
share live.yml:

* Artifact split: the JSON verdict reports are tiny and useful on
  green runs too (flaky trends), so they now upload on every run as
  live-reports-<category>. The service/worker logs, which can reach
  hundreds of MB after a multi-hour run, stay failure-only as
  live-logs-<category>.

* Nightly visibility: a failed scheduled run only emails the workflow
  author. A report-nightly job now mirrors the state into a
  live-nightly tracking issue: opened on first failure, commented on
  repeats, closed automatically on the next green run. A suite skipped
  because the plan job died counts as a failure; a cancelled run
  reports nothing.
Document the per-job verdict summaries, the always-uploaded
live-reports-<category> artifact, and the live-nightly tracking issue.
Nothing has invoked `turbo run test:live` (or the contracts test:live
script it wrapped) since the unified runner replaced the root script
in f751932. Keeping them left a second, guard-less entry point that
skipped the stack reset, lock, and flake verdict. The runner is now
the single way to run live tests.
Record the design decision in the runner header: turbo models a DAG
of stateless, cacheable tasks, while a live run needs stateful
orchestration (two-round flake verdict, docker lifecycle against one
shared node, ZK-key self-heal, infra-vs-test exit codes). Turbo still
handles the DAG-shaped steps (compile, harness smoke) inside the run.
The harness PR bundled a `compact`->`compile` and `fmt-and-lint`->`check`
script rename that is unrelated to the test harness. Per review, back it
out so this PR stays scoped to the harness. The rename will land as its
own PR with a changelog entry.

* package.json / contracts/package.json / turbo.json: restore the
  `compact:*` and `fmt-and-lint*` names, keeping every harness script,
  turbo task, and dependency
* bug_report.yml / checks.yml / test.yml: reverted to main
* scripts/test-live.ts, scripts/keyIntegrity.ts: invoke `yarn compact`
  again
Co-authored-by: Andrew Fleming <fleming-andrew@protonmail.com>
Signed-off-by: 0xisk <0xisk@proton.me>
The live CI workflow (`live.yml`) and its "Live tests in CI"
CONTRIBUTING subsection are too large and complex to review alongside
the harness. Per review, remove them here and land them in a separate
PR stacked on this branch.

Refs: #673
Companion to the previous commit: the "Live tests in CI" subsection
documented and linked `live.yml`, which moves to the follow-up PR.

Refs: #673
Compiling the archive module as part of the full `compact` task fails
locally (per review). Archive is excluded from the full compile on
`main` too. Keep the `compact:archive` task for targeted runs, but drop
it from the aggregate `dependsOn`.

Refs: #673
`docker compose up -d --wait` waits on every service, including
proof-server, which has no healthcheck. That is not portable across
Docker Compose versions and aborts `yarn test:live` on some setups
(reported in review). Start all services, then wait only on node and
indexer, which expose healthchecks.

Refs: #673
The shielded preset rename (`ShieldedMultiSig*` to capability-based
names) is a user-facing contract change unrelated to the test harness.
Per review, revert it here so this PR keeps the presets under their
existing names; the rename lands as its own stacked PR with a changelog
entry.

Harness-only content changes are preserved (e.g. the block-limit
circuit trim in ShieldedMultiSig.compact).

Refs: #673
Per review, enable Corepack once as a prerequisite instead of prefixing
every command with `corepack`. CI already runs `corepack enable`, so
plain `yarn` resolves to the version pinned in package.json.

* CONTRIBUTING: document the one-time `corepack enable`; drop the prefix
  from every command
* scripts/test-live.ts, keyIntegrity.ts, live.globalSetup.ts: invoke and
  print plain `yarn`

Refs: #673
The live runner already scopes to a category via a positional arg
(`test:live <category> [filter]`), and CI invokes it that way
(`yarn test:live "$CATEGORY"`). Drop the per-category
`test:live:multisig` script and document the argument form instead, so
local usage matches CI and the script list does not grow one entry per
category.

Refs: #673
Round 2 reset the shared node once, then re-ran all failed files with a
single worker. That removes parallel collisions but not sequential state
pollution: an earlier round-2 file can dirty the node (e.g. leave a spent
coin -> "Custom error: 103") and fail a later one, misclassifying a flake
as REAL. Reset the node before each failed file so every one runs alone
on a fresh node, matching the "passes in isolation" intent. Round 2 only
re-runs the failed files, so the extra resets are cheap.

Refs: #673
Per review, wrap the happy-path burn tests (real spends, dry-only for
now) in a single `describe.skipIf(isLiveBackend())('happy path (dry
only)')` instead of tagging each with `it.skipIf`, matching the
describe-based split already used in ShieldedTreasuryStateless. The
backend-agnostic reject tests stay as direct children; mint keeps its
lone skipIf (a single test needs no wrapper).

Refs: #673
The preset dropped getProposalRecipient/Amount/Color and getReceivedMinusSent
to fit the deploy block limit. The preset simulator re-added them as methods
that read like circuit calls but only derived the values, which is misleading.
Remove them and read the fields directly off getProposal and the received/sent
totals the preset does expose.

Refs: #673
Extend the tracker tests beyond the dry surface: resolve a wallet- and a
contract-owned coin to its mt_index, the anchor-block re-read, the not-found
failure, and the process-wide singleton. Only the indexer event stream is
mocked; the commitment math runs for real, so a resolve proves the tracker
computes the commitment the indexer would emit.

Refs: #673
The port() guard's promise (an empty, non-numeric, or out-of-range override
throws at load time, naming the env var) had no test. Add the three cases.

Refs: #673
sort() hid the build order, so the test could not tell serial from concurrent
building. Assert the exact order and a max concurrency of one via a counting
builder. Also fix the class doc that wrongly said "concurrently"; the builder
is serial on purpose (it funds empty signers from the shared deployer).

Refs: #673
test:live matches a filename substring, so a name that prefixes others runs
them all. Stop advertising a single-file run; say any file matching the name.

Refs: #673
Co-authored-by: Andrew Fleming <fleming-andrew@protonmail.com>
Signed-off-by: 0xisk <0xisk@proton.me>
Rebasing onto main replays this branch's earlier compile->compact and
check->fmt-and-lint rename churn on top of main's #680 rename, which
silently reverts the script and Turbo task names back to the old form
wherever git saw no conflict.

Restore main's compile:*/lint* naming across package.json, turbo.json,
the test/bug-report workflows, and scripts/keyIntegrity.ts, and refresh
yarn.lock for main's @types/node and fast-check bumps (#685, #686).
@0xisk
0xisk force-pushed the test/multisig-live-harness branch from a45bda7 to 7f6e09e Compare July 17, 2026 16:15
@0xisk
0xisk enabled auto-merge (squash) July 17, 2026 16:15
@0xisk

0xisk commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

@0xisk just a heads up, there are still conflicts. And I assume the extra commits after the last review are from force pushing something? The conflicts were (and are) in yarn.lock

@andrew-fleming I fixed the conflicts with rebase and force push with the main.

@0xisk
0xisk requested a review from andrew-fleming July 17, 2026 16:19

@andrew-fleming andrew-fleming left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@0xisk
0xisk merged commit dd88196 into main Jul 18, 2026
9 checks passed
@0xisk
0xisk deleted the test/multisig-live-harness branch July 18, 2026 03:52
0xisk added a commit that referenced this pull request Jul 20, 2026
Run the live suite separately from the required PR checks: nightly on
`main`, on-demand dispatch, and via the `live-tests` PR label. A plan
job reads the live-ready categories from the runner and fans out one
job per category, each with its own stack and job budget. Documented
in the "Live tests in CI" CONTRIBUTING subsection.

Stacked on the live-harness branch (#673); rebases onto main once that
merges.

Refs: #673
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dev: run multisig shielded specs on the live harness dev: core live-backend harness infrastructure

2 participants