fix(desktop): supervise and re-arm relay-mesh runtime#2823
Open
micspiral wants to merge 9 commits into
Open
Conversation
Relay-mesh agents went silent with no error if the embedded mesh runtime exited or wedged after launch: `mesh_llm_runtime` stayed `Some` while the local OpenAI ingress on :9337 was dead, so `ensure_relay_mesh_for_record` took the "already running" fast path and `wait_for_mesh_inference` just timed out against a dead endpoint (#2062). Add a fast liveness probe (`mesh_ingress_is_live`: single 3s `GET /v1/models`, the same call the issue used to confirm the ingress was down). When a runtime handle is present but the ingress is unreachable, drop the stale runtime (best-effort stop; never block re-arm on a wedged runtime — the zombie-guard motivation) and fall through to re-arm it via the normal bootstrap path. Resolves the reported symptom (option 1 + zombie guard from the issue): long-lived sessions recover shared-compute agents on the next dispatch instead of staying silent until a manual restart. Additive; the healthy fast path (live ingress) is unchanged. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Brad #2304: mesh_ingress_is_live inside ensure_relay_mesh_for_record only ran on start/restore — not after launch. Local agents hit :9337 themselves, so there is no desktop turn-dispatch hook; add a bounded coordinator watchdog (15s base, double on failure to 120s) that probes ingress, drops a zombie runtime handle, and re-arms via ensure_relay_mesh_for_record for relay-mesh agents. Failed re-arm writes an actionable \"Buzz shared compute offline\" last_error. Share drop_stale_mesh_runtime_if_ingress_dead with ensure path. Unit tests cover dead-port probe, noop drop without handle, and failure copy. Refs #2062 Signed-off-by: Bartok9 <danielrpike9@gmail.com>
…opped agents Addresses Brad's #2304 review of head 6d20d20: 1. Bounded stale stop() — a wedged runtime could hang stop().await and block the watchdog forever, defeating the never-block-re-arm intent. Wrap in a 3s tokio::time::timeout (matches probe budget); log and drop the handle on timeout/error so the watchdog keeps making progress. 2. Probe/evict race — capture the runtime identity (new monotonic DesktopMeshRuntime::id) before the ingress probe .await and only evict if the same handle is still installed on lock reacquire, so a concurrent stop/start replacement is never evicted. 3. Stopped/manual records — re-arm now filters to local relay-mesh agents whose own process is actually running (runtime_pid + process_is_running), so deliberately stopped agents are not resurrected. Adds unit tests for the running/stopped/non-mesh re-arm target filter. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Address remaining review gaps on the watchdog re-arm path: 1. Keep bounded stop timeout (already present) and document the never-block-re-arm invariant. 2. Extract identity-compare helper + injectable ingress probe so probe/evict never drops a concurrent replacement runtime. 3. Intersect relay-mesh records with live managed_agent_processes pubkeys (not every configured mesh record / pid-only heuristic). 4. persist/clear mesh last_error under managed_agents_store_lock, bump updated_at, preserve unrelated errors, surface save failures. Tests: process-map filter, identity skip, stop budget, error classifier; hardware-gated kill-:9337 recovery documented as Signed-off-by: Bartok9 <danielrpike9@gmail.com> #[ignore] for manual mesh machines.
…micspiral #2304) Address the mesh-applicability review on the re-arm watchdog: 1. Consecutive-failure debounce before eviction (merge-blocker). A single dead /v1/models probe past the 3s budget (transient stall: model load/reload, VRAM alloc, GC/mmap pause, inference saturation) no longer cold-restarts a healthy runtime. Require DEAD_PROBE_EVICT_THRESHOLD (2) consecutive dead probes on the watchdog cadence; a live probe or an identity-mismatch replacement resets the streak. Counter lives on AppState (AtomicU32). 2. clear_mesh_last_error_if_set now matches a sentinel prefix (MESH_REARM_ERROR_SENTINEL) instead of the loose "shared compute" substring, so recovery only clears errors this watchdog set. 3. Documented the serve->client re-arm mode transition as intentional-by-design (safe fail-safe; serve restoration stays restore_mesh_sharing's job). Tests: added eviction_debounces_transient_dead_probe; updated error classifier + actionable-copy tests for the sentinel. 17 passed, 2 ignored. Signed-off-by: Bartok9 <danielrpike9@gmail.com>
CI on #2304 failed: cargo fmt --check on mesh_llm paths, and desktop file-size gate (app_state + mesh_llm growth for watchdog). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
The AppState field is #[cfg(feature = "mesh-llm")] but its initializer was unconditional, breaking non-mesh builds under -D warnings. Gate the initializer and fully-qualify AtomicU32 so the import is not left unused (micspiral, #2304). Signed-off-by: Bartok9 <danielrpike9@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2062
This carries forward the relay-mesh recovery work from #2304 by @Bartok9 (cherry-picked with original authorship/sign-offs) and adds the startup/readiness supervision and packaging fixes found while validating it against a real two-machine Buzz setup.
From #2304
:9337) after launch and re-arm a stale relay-mesh runtime.Added here
:9337), rather than management port:3131, as Buzz's client-readiness boundary. A usable client no longer fails or holds agent-save open merely because management startup is still pending.buzz-acp,buzz-agent, and tool sidecars to be reported as missing.Validation
Automated:
just ci— passed, including formatting, Clippy with warnings denied, desktop/web/mobile checks and tests, and builds.-D warnings— passed.mesh-llm— built successfully; every bundled sidecar passed executable-mode and deep code-signature verification.Live two-machine E2E:
unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M.buzz-agent/relay-mesh agent.:9337accepted inference and the saved ACP harness started with no error while:3131was still unavailable.CORRECTED-MINI-E2E-OK).:3131was still unavailable.AFTER-RESTART-E2E-OK).@C55message traversed desktop →buzz-acp→buzz-agent→ mini:9337→ M5 compute and published the requested reply successfully.