PRISM Compute Plane: miner-funded GPU worker plane (base)#37
Draft
echobt wants to merge 11 commits into
Draft
Conversation
Add src/base/compute/ package: - provider.py: ProviderClient protocol, InstanceSpec (mandatory max_lifetime_hours/max_price_per_hour bounds), Offer/Instance, typed ProviderError/CostGuardrailError. - lium.py: LiumClient over httpx (X-API-Key, base https://lium.io/api). list_offers price filtering; provision refuses unbounded/over-priced specs before any network call, always sends bounded termination_hours, and terminates+verifies on any post-rent failure (try/finally); idempotent ensure_ssh_key/ensure_template; idempotent terminate; verify_terminated via GET /pods; stream_logs; watchtower digest; GET /users/me balance. API key never logged/repr'd/in errors. 50 offline respx unit tests (VAL-PROV-001/003/004/005/011/017/018 + secret hygiene). Full base gate green: 1316 passed, coverage 89.1%.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…-client secret hygiene
Add build_lium_worker_template (Lium CustomTemplateRequest payload) and build_targon_worker_app (Targon app definition) in src/base/compute/ worker_deployment.py. Both pin the docker image BY DIGEST via a shared pinned_image_reference helper; the image+digest default to the published prism-evaluator digest (M1 placeholder) and are inputs so M2 swaps in docker/Dockerfile.worker with a one-line change. Well-formedness is enforced (required fields, internal ports incl. 22, env plumbing, is_private, no embedded secrets) and covered by offline respx tests that also assert the whole compute suite makes zero real network calls under respx strict mode and needs no provider credentials (VAL-PROV-009/010/016).
…TESTS Add scripts/live_lium_e2e.py driving the M1 provider clients against production: read-only reachability (Lium users/me + executors + watchtower/digest, Targon inventory + apps) plus one batched Lium rental cycle (ensure ssh key + template -> rent cheapest suitable executor -> poll RUNNING -> ssh nvidia-smi -> logs -> DELETE -> verify gone -> record balance delta). Pod is deleted in a finally on every path; opt-in via BASE_LIVE_PROVIDER_TESTS=1 so the default suite stays offline.
…gpu shape - LiumClient.provision rejects sub-1-hour max_lifetime_hours so termination_hours never truncates to 0 (auto-termination stays on) - widen post-rent cleanup guard to key off rent success, not pod-id resolution: a transient GET /pods failure during resolution still best-effort terminates + verifies the just-rented pod - normalize Targon Offer.price_per_hour to per-GPU (cost_per_hour / gpu_count) so the per-GPU max_price cap filters multi-GPU shapes correctly - default WORKER_GPU_SHAPE 'h100' -> live-valid 'h100-small'
Add the miner-funded GPU worker plane registry (architecture.md sec 3.3),
gated behind compute.worker_plane_enabled:
- alembic 0009: worker_registrations + worker_faults + worker_request_nonces
on the existing chain (0008 -> 0009), no legacy table altered.
- WorkerCoordinationService mirroring validator_coordination patterns:
POST /v1/workers/register (sr25519 miner binding verified against the mock
metagraph, binding-nonce replay protection, no silent cross-owner rebind),
POST /v1/workers/{id}/heartbeat, GET /v1/workers (fleet: status/owner/
provider/last-seen/faults, authenticated-but-not-admin), and
GET /v1/workers/active?hotkey= (admission surface).
- Lifecycle pending -> active -> stale -> retired with
compute.worker_heartbeat_ttl_seconds (default 120); retired is terminal
(no heartbeat resurrection); staleness derived on read + a background pass.
- worker_auth: binding message/verify, signed-request verifier + registered-
worker/validator eligibility, worker nonce store.
- Wired into create_proxy_app + the master proxy CLI (flag-off => unmounted).
Unit + Postgres integration tests (15433) cover VAL-MASTER-001/002/015/016/
018/021 and VAL-AGENT-015.
Add the miner-funded WorkerAgent (src/base/worker/): register under a
miner-signed binding, heartbeat to stay active, pull gpu-only replicas,
execute via the AssignmentExecutor seam on its local broker, and post
results that always carry an ExecutionProof envelope (sr25519 over
sha256('{manifest_sha256}:{unit_id}'), pinned identically to prism).
Extract the shared agent-loop primitives into base/coordination/agent_loop
(BackoffPolicy, is_transient_error, AgentCycleSummary, sleep_until,
backoff_sleep); the validator agent now imports them with behavior unchanged.
Add the master worker assignment plane (worker_assignments table + 0010
migration, worker-authenticated pull/result routes gated on registration
and liveness, never a validator permit) and wire it behind
compute.worker_plane_enabled.
Fulfills VAL-AGENT-002/003/004/005/006/007/008/016/017/018.
Add a top-level `base worker` Typer app distinct from the legacy `base master worker` Swarm group: - deploy --provider local starts a miner-funded agent against a local master and reports it active within 60s. - deploy --provider lium|targon requires the provider key env (actionable refusal before any network), bounds offer selection by --max-price preferring an exact gpu_count executor (next-cheapest fallback), and never transmits the provider key to the master (pod env excludes it). - status renders the fleet from GET /v1/workers (signed as the worker key). Thread startup_commands through InstanceSpec, LiumClient.ensure_template and build_lium_worker_template (validated metachar-free) per the live-learned Lium rent constraint; keep docker/Dockerfile.worker's exec-form entrypoint metachar-free. Add WorkerSettings, worker/miner keypair resolvers, signed list_workers, config/worker.example.yaml, and unit tests. All behind compute.worker_plane_enabled; full base gate green.
Add WorkerAssignmentEngine that materializes gpu work-unit replicas onto ACTIVE distinct-owner workers behind compute.worker_plane_enabled: R=2 with self-evaluation exclusion (unit waits under sole-capacity scarcity), graceful degradation to R=1 with a recorded warning, per-worker gpu concurrency 1, and per-replica deadline/reassignment bounded by max_attempts. The validator AssignmentService skips worker-plane capabilities when the flag is on, so flag-OFF gpu routing to validators stays byte-identical to legacy.
… behind flag Reconcile replicated gpu worker results (architecture.md 3.3): matching ExecutionProof.manifest_sha256 forwards exactly one result to the challenge; divergent hashes dispute the unit (never forwarded, before or after audit) and create a validator-executor audit unit whose outcome writes worker_faults for the divergent worker(s), visible in fleet status. Single-replica reporting terminates deterministically (accept-after-degrade with warning); late/foreign posts stay rejected with replica state intact. All gated by compute.worker_plane_enabled (reconciler is None when off).
Legacy AssignmentService.reclaim_stale_assignments treated a null assigned_validator_hotkey as 'offline validator => reassignable', so under the worker plane a worker-owned prism PRIMARY (ASSIGNED, null hotkey by design) churned back to PENDING every MasterOrchestrationDriver.run_once pass. _assign_pending_in_session already skipped such units via worker_plane_capabilities; _reclaim_in_session did not. Factor the guard into AssignmentService._worker_plane_owns and apply it in BOTH paths so a worker-owned primary is neither assigned nor reclaimed here, while a genuinely stale validator unit (cpu, or gpu AUDIT with executor_kind= validator) is still reclaimed. Flag OFF keeps reclaim byte-identical to legacy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mission branch for the PRISM Compute Plane (base repo). Delivered incrementally per milestone; this PR aggregates the mission work on `mission/compute-plane`.
M1 providers (in progress)
compute-provider-contract-and-lium-client
All new behavior is additive (new package); no existing behavior changed. Full base gate green: ruff, format, mypy, and pytest (1316 passed, coverage 89.1%).