Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
2b733ee
feat(compute): provider contract + Lium client with cost guardrails
echobt Jul 6, 2026
662c9c2
feat(compute): Targon client with typed credit/balance errors + cross…
echobt Jul 6, 2026
a0b23a1
feat(compute): worker-image deploy definitions pinned by digest
echobt Jul 6, 2026
7b5084e
test(compute): live Lium/Targon E2E script behind BASE_LIVE_PROVIDER_…
echobt Jul 6, 2026
b9e2e46
fix(compute): harden Lium guardrails, Targon per-GPU pricing, worker …
echobt Jul 6, 2026
2f43562
feat(master): worker registry + miner-signed enrollment surface
echobt Jul 6, 2026
4eb06ad
feat(worker): worker agent runtime with proof-carrying results
echobt Jul 7, 2026
c260511
feat(worker): base worker CLI (agent|deploy|status) + worker image
echobt Jul 7, 2026
16ba7ae
feat(master): worker-plane gpu assignment engine behind flag
echobt Jul 7, 2026
232b2d6
feat(master): worker-plane reconciliation, disputes, and audit faults…
echobt Jul 7, 2026
35d0b8c
fix(master): apply worker-plane guard symmetrically in reclaim path
echobt Jul 7, 2026
7637eaa
feat(master): dual-auth admission fleet-read via internal bridge bearer
echobt Jul 7, 2026
ee5579c
feat(mission): local cross-repo e2e harness + fleet-status fault column
echobt Jul 7, 2026
b9f18a7
feat(master): signed read-only /v1/workers/units for dispute discovery
echobt Jul 7, 2026
808341d
docs(worker-plane): miner worker deploy guide (Lium+Targon) + README …
echobt Jul 7, 2026
71fb72f
test(mission): flags-OFF legacy regression harness (VAL-CROSS-006)
echobt Jul 7, 2026
e423bac
test(mission): live Lium worker E2E deploy->enroll->execute->prove->t…
echobt Jul 7, 2026
cfde517
fix(worker): unblock real Lium deploy (WAF loopback + required public…
echobt Jul 7, 2026
f3c632d
fix(compute): Targon two-step create-then-deploy + sub-1h cost guard
echobt Jul 7, 2026
340ef0c
test(supervisor): harden image-updater caplog tests against global lo…
echobt Jul 7, 2026
306b527
fix(master): key worker-unit faults by (challenge_slug, work_unit_id)…
echobt Jul 7, 2026
9c4f2dd
fix(compute): arm rent cleanup on unparseable body + Targon per-GPU c…
echobt Jul 7, 2026
86283e7
merge: integrate origin/main into mission/compute-plane
echobt Jul 7, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Docs are grouped by audience.
**Miners**

- [Miner guide](docs/miner/README.md) — choose a challenge, submit through the proxy, and track leaderboards.
- [Miner worker deployment guide](docs/miner/worker-plane.md) — deploy a miner-funded GPU worker on Lium or Targon with `base worker deploy`, monitor it with `base worker status`, and understand the PRISM admission rule and proof tiers.

**Validators / operators**

Expand Down Expand Up @@ -214,6 +215,56 @@ capabilities, subscriptions, last heartbeat, resolved identity); it NEVER expose

---

## Miner-Funded GPU Worker Plane

The **worker plane** moves PRISM's heavy GPU evaluation off BASE validators and onto **worker
agents running in GPU instances the miners themselves fund** (rented on Lium or Targon, or their
own hardware). Validators keep only light plausibility checks, probabilistic replay audits, and
on-chain weight submission. PRISM scoring is unchanged. Everything is gated behind
`compute.worker_plane_enabled` (env `BASE_COMPUTE__WORKER_PLANE_ENABLED`); with the flag **off**
(the default) gpu units route to validators exactly as today and the worker coordination surface is
not even mounted (byte-for-byte legacy behavior).

- **`base worker` CLI** (top-level, distinct from the legacy `base master worker` Swarm group):
- `base worker deploy --provider lium|targon|local [--max-price <usd/gpu/hr>]` — provision a
worker on a miner-funded instance (or locally) and enroll it. The miner's provider key
(`LIUM_API_KEY` / `TARGON_API_KEY`) authenticates provider calls only and is **never** sent to
the master. Deploy always sets a bounded instance lifetime and refuses offers above the price
cap.
- `base worker agent` — run the long-lived agent loop (register → heartbeat → pull → execute →
post), authenticated as the worker keypair.
- `base worker status [--hotkey <miner-hotkey>]` — render the fleet from the master's
`GET /v1/workers` (status, owner, provider, last-seen, fault count).
- **Signed enrollment**: the miner signs `worker-binding:{worker_pubkey}:{miner_hotkey}:{nonce}`
(sr25519); the master verifies it against the metagraph. Status lifecycle is
`pending → active → stale → retired`; `active` requires a verified binding **and** a heartbeat
within `compute.worker_heartbeat_ttl_seconds` (default 120).
- **Admission rule**: when enforced, a miner must have ≥1 **active** worker bound to their hotkey
to submit to PRISM; otherwise the submission is rejected `403 NO_ACTIVE_WORKER`. This is the
incentive to supply GPU without changing the subnet economics.
- **Anti-collusion assignment**: a worker never evaluates its own owner's submission; each gpu unit
is replicated across **R=2 distinct-owner** workers (`compute.replication_factor`, degrading to
1 with a recorded warning). The master reconciles by comparing each replica's
`ExecutionProof.manifest_sha256`: equal ⇒ one result forwarded to the challenge; divergent ⇒ the
unit is `disputed` and a validator **audit** unit replays it authoritatively, flagging the lying
worker via `worker_faults`.
- **ExecutionProof tiers** (carried on every result):
- **Tier 0** (mandatory, all backends): deterministic manifest hash + worker sr25519 signature —
the source of truth for reconciliation and audits.
- **Tier 1**: evaluator image digest matches the pinned digest (on Lium, cross-checkable against
the provider's signed image-digest endpoint).
- **Tier 2**: in-guest TDX + nvtrust attestation — schema shipped but gated **off on Targon**
today (no consumer-facing attestation surface), so Targon proofs carry tier ≤ 1.
- Audit sampling is tier-modulated (tier 0 ≈ 10%, tier 1 ≈ 5%, tier 2 ≈ 2%).

Miners: see the [Miner worker deployment guide](docs/miner/worker-plane.md) for prerequisites,
exact Lium/Targon commands, cost guidance, monitoring, and troubleshooting. The master subsystems
live under `src/base/worker/` (agent) and `src/base/master/worker_coordination.py` (registry,
enrollment, fleet, admission), with provider clients in `src/base/compute/`
(`lium.py` / `targon.py`).

---

## What BASE Does

BASE coordinates the full lifecycle of a multi-challenge subnet:
Expand Down Expand Up @@ -252,6 +303,18 @@ master's final normalized vector from the weights API and commits it under its o
dedicated submit-only host can run the same submitter). The master aggregates but never submits
on-chain. Challenge API services are run by the master (manager) node.

### Workers (miner-funded GPU worker plane)

Workers are **miner-funded GPU executors** for PRISM. A miner runs `base worker deploy` to launch a
worker agent inside a GPU instance the miner pays for (rented on **Lium** or **Targon**, or their
own hardware); the agent registers with the master under a miner-signed hotkey↔worker binding,
heartbeats, pulls `gpu` work units, executes the PRISM evaluator on its own broker, and posts
results carrying an `ExecutionProof`. The provider API key (`LIUM_API_KEY` / `TARGON_API_KEY`)
stays in the miner's environment and is never sent to the master. The entire worker plane is gated
behind `compute.worker_plane_enabled` (env `BASE_COMPUTE__WORKER_PLANE_ENABLED`, default off ⇒
byte-for-byte legacy behavior). See the [Worker plane](#miner-funded-gpu-worker-plane) section and
the [Miner worker deployment guide](docs/miner/worker-plane.md).

---

## Repository Layout
Expand Down
1 change: 1 addition & 0 deletions alembic/versions/0005_create_work_assignments.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
"running",
"completed",
"failed",
"disputed",
name="work_assignment_status",
native_enum=False,
)
Expand Down
185 changes: 185 additions & 0 deletions alembic/versions/0009_create_worker_registry.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
"""Create miner-funded GPU worker registry tables.

Adds the worker-plane storage (architecture.md sec 3.3): ``worker_registrations``
(the enrolled workers + their ``pending`` -> ``active`` -> ``stale`` -> ``retired``
lifecycle), ``worker_faults`` (reconciliation/audit fault attribution surfaced in
the fleet view), and ``worker_request_nonces`` (replay protection for the miner
binding nonce and the signed request-envelope nonce). No pre-existing table is
altered.

Revision ID: 0009_create_worker_registry
Revises: 0008_validator_subscriptions
Create Date: 2026-07-06 00:00:00.000000
"""

from __future__ import annotations

from collections.abc import Sequence

import sqlalchemy as sa

from alembic import op

revision: str = "0009_create_worker_registry"
down_revision: str | None = "0008_validator_subscriptions"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None

worker_status = sa.Enum(
"pending",
"active",
"stale",
"retired",
name="worker_status",
native_enum=False,
)


def upgrade() -> None:
"""Apply the migration."""

op.create_table(
"worker_registrations",
sa.Column("id", sa.Uuid(as_uuid=True), nullable=False),
sa.Column("worker_id", sa.Text(), nullable=False),
sa.Column("worker_pubkey", sa.Text(), nullable=False),
sa.Column("miner_hotkey", sa.Text(), nullable=False),
sa.Column("binding_signature", sa.Text(), nullable=False),
sa.Column("binding_nonce", sa.Text(), nullable=False),
sa.Column("provider", sa.Text(), nullable=False),
sa.Column("provider_instance_ref", sa.Text(), nullable=True),
sa.Column("capabilities", sa.JSON(), server_default="[]", nullable=False),
sa.Column("status", worker_status, server_default="pending", nullable=False),
sa.Column("last_seen_meta", sa.JSON(), server_default="{}", nullable=False),
sa.Column("last_heartbeat_at", sa.DateTime(timezone=True), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.Column(
"updated_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_worker_registrations")),
sa.UniqueConstraint("worker_id", name="uq_worker_registrations_worker_id"),
sa.UniqueConstraint(
"worker_pubkey", name="uq_worker_registrations_worker_pubkey"
),
)
op.create_index(
"ix_worker_registrations_status",
"worker_registrations",
["status"],
unique=False,
)
op.create_index(
"ix_worker_registrations_miner_hotkey",
"worker_registrations",
["miner_hotkey"],
unique=False,
)
op.create_index(
"ix_worker_registrations_last_heartbeat_at",
"worker_registrations",
["last_heartbeat_at"],
unique=False,
)
op.create_index(
"ix_worker_registrations_status_miner_hotkey",
"worker_registrations",
["status", "miner_hotkey"],
unique=False,
)

op.create_table(
"worker_faults",
sa.Column("id", sa.Uuid(as_uuid=True), nullable=False),
sa.Column("worker_id", sa.Text(), nullable=False),
sa.Column("work_unit_id", sa.Text(), nullable=False),
sa.Column("challenge_slug", sa.Text(), nullable=True),
sa.Column("detail", sa.Text(), nullable=True),
sa.Column(
"created_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_worker_faults")),
)
op.create_index(
"ix_worker_faults_worker_id",
"worker_faults",
["worker_id"],
unique=False,
)
op.create_index(
"ix_worker_faults_work_unit_id",
"worker_faults",
["work_unit_id"],
unique=False,
)

op.create_table(
"worker_request_nonces",
sa.Column("id", sa.Uuid(as_uuid=True), nullable=False),
sa.Column("hotkey", sa.Text(), nullable=False),
sa.Column("nonce", sa.Text(), nullable=False),
sa.Column("body_hash", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id", name=op.f("pk_worker_request_nonces")),
sa.UniqueConstraint(
"hotkey", "nonce", name="uq_worker_request_nonces_hotkey_nonce"
),
)
op.create_index(
"ix_worker_request_nonces_created_at",
"worker_request_nonces",
["created_at"],
unique=False,
)
op.create_index(
"ix_worker_request_nonces_hotkey",
"worker_request_nonces",
["hotkey"],
unique=False,
)


def downgrade() -> None:
"""Revert the migration."""

op.drop_index(
"ix_worker_request_nonces_hotkey",
table_name="worker_request_nonces",
)
op.drop_index(
"ix_worker_request_nonces_created_at",
table_name="worker_request_nonces",
)
op.drop_table("worker_request_nonces")

op.drop_index("ix_worker_faults_work_unit_id", table_name="worker_faults")
op.drop_index("ix_worker_faults_worker_id", table_name="worker_faults")
op.drop_table("worker_faults")

op.drop_index(
"ix_worker_registrations_status_miner_hotkey",
table_name="worker_registrations",
)
op.drop_index(
"ix_worker_registrations_last_heartbeat_at",
table_name="worker_registrations",
)
op.drop_index(
"ix_worker_registrations_miner_hotkey",
table_name="worker_registrations",
)
op.drop_index(
"ix_worker_registrations_status",
table_name="worker_registrations",
)
op.drop_table("worker_registrations")
Loading
Loading