diff --git a/CLAUDE.md b/CLAUDE.md index e39051e5..0fe15984 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -167,6 +167,8 @@ Codeman is a Claude Code session manager with web interface and autonomous Ralph **Cron (cron-style `CronJob`s)**: saved, named jobs with a recurring schedule (`once`/`interval`/`daily`/`weekly`), enable/disable, Run Now, next-run calc, and per-job run history (`CronJobRun`). ⚠️ **Distinct from the legacy `ScheduledRun`** (`/api/scheduled`, a run-now duration-bounded autonomous loop) — the two never interact; the legacy concept keeps the `Scheduled*` names, the recurring-job feature is `Cron*`. `CronService` (`src/cron/cron-service.ts`) owns CRUD + the 30s background due-tick (`tickDueJobs`, registered via `cleanup.setInterval` in `server.ts`; `init()` recomputes nextRunAt on boot) and **reuses the existing session layer** (create → `addSession` → `setupSessionListeners` → `startInteractive`/`startShell` → prompt via `writeViaMux`/`write`) rather than rebuilding tmux logic. Next-run math is pure/unit-tested in `cron-time.ts` (SERVER-LOCAL timezone for daily/weekly). Dup-launch guard = `lastDueKey` (jobId:fireTime); schedule is advanced BEFORE launch so a slow launch can't re-trigger. `once` jobs self-disable after firing (`completedOnce`). Persisted via `AppState.cronJobs`/`cronJobRuns` (StateStore accessors). Routes `/api/cron/jobs*` + `/api/cron/runs` (`cron-routes.ts`, `CronPort`); schema `CronJobSchema` (cross-field `superRefine`; the `.partial()` update schema does NOT re-run it); SSE `cron:*`. Frontend `cron-ui.js` (#cronModal). Claude/shell/opencode/codex/gemini agent types. Tests: `test/cron-time.test.ts`, `test/cron-service.test.ts`. Design: `docs/cron-discovery.md`. +**Remote sessions (SSH)**: Sessions can run the agent inside a durable `tmux -L codeman-remote new-session -A` **on a remote host** so it survives the SSH drop (COD-104), and can also **discover + attach** to `codeman-*` sessions another Codeman launched there — attached (`owned:false`) sessions **detach, never kill** on tab close (COD-105). **Shared/collaborative** (COD-106): remote set-options are scoped per-session (never `-g`) and `window-size latest` lets multiple clients attach the same session at different viewports without clamping to the smallest; a client count surfaces a "shared · N" badge. **Auto-reconnect** (COD-108): a bounded-backoff watcher re-establishes a dropped remote session's local ssh pane and reattaches the still-running durable remote tmux (kill-switch `remoteAutoReconnect`, default ON). Owned sessions propagate `kill-session` to the remote on close; non-owned never do. ⚠️ Command-injection surface (COD-107): all ssh command lines flow through the single shell-safe `buildSshConnectionArgs()` — every user field (`-J jumpHost`, `-i identity`, `-o`) is `shellescape`d; never hand-build an ssh line elsewhere. Full design: `docs/remote-sessions.md`. + **External CLI modes (OpenCode, Codex, Gemini)**: `isExternalCliMode()` in `session.ts` (`mode === 'opencode' || 'codex' || 'gemini'`) gates Claude-specific behavior — Ralph tracker, BashToolParser, token/CLI-info parsing, and ❯-prompt readiness detection are all skipped (these CLIs render their own TUIs; readiness = output stabilization instead). All three modes **require tmux — no direct PTY fallback** — because secrets are injected via `tmux setenv` (socket-scoped `${this.tmux()} setenv`, never on the spawn command line): OpenCode gets `OPENCODE_CONFIG_CONTENT` etc., Codex gets `OPENAI_API_KEY`/`CODEX_API_KEY`/`CODEX_HOME` (`setCodexEnvVars`), Gemini gets `GEMINI_API_KEY`/`GOOGLE_API_KEY`/`GOOGLE_CLOUD_PROJECT`/`GOOGLE_APPLICATION_CREDENTIALS`/`GOOGLE_GENAI_USE_VERTEXAI` etc. (`setGeminiEnvVars`, all in `tmux-manager.ts`). Codex specifics: command built by `buildCodexCommand()` (`--model`, `resume `, `--dangerously-bypass-approvals-and-sandbox` from the `codexConfig` payload / `codexDangerouslyBypassApprovals` app setting; `renderMode` is schema-coerced to `'hybrid'`, the only supported mode). Gemini specifics: command built by `buildGeminiCommand()` (`--skip-trust` always, `--approval-mode ` defaulting to `yolo` for parity with Claude's `--dangerously-skip-permissions`, `--model`, `--resume` from the `geminiConfig` payload); availability via `GET /api/gemini/status` — session/quick-start routes fail with `OPERATION_FAILED` + install hint (`npm install -g @google/gemini-cli`) when missing. Codex AND Gemini export `COLORTERM=truecolor` + unset `NO_COLOR` (other modes unset `COLORTERM`); Gemini joins `isAltScreenStripMode()` (Codex/Claude/Gemini are Ink TUIs that repaint inline → strip alt-screen/`3J` so scrollback survives). Codex availability via `GET /api/codex/status`. Frontend: run-mode dropdown → `runCodex()`/`runGemini()` in `session-ui.js` ("Run CX"/"Run GM" labels), App Settings → Codex CLI tab; Respawn/Ralph options are Claude-only, so session options open on the Summary tab for external CLI sessions. ⚠️ `run*()` MUST unwrap the `{success,data}` envelope (`(await res.json()).data.available` / `data.data.sessionId`) — reading the raw shape silently breaks the run. Tests: `test/run-mode-ui.test.ts` + `test/gemini-mode.test.ts` (vm-sandbox harness, no real DOM). **Remote SSH cases** (COD-94/#145): cases can point at a **remote host** (`~/.codeman/remote-hosts.json` + `remote-cases.json` via `src/remote-hosts.ts`; CRUD under `/api/cases` — cases route file). A remote session launches a LOCAL tmux pane running `ssh ` that creates a durable REMOTE tmux session on a **dedicated socket** `-L codeman-remote` with name `codeman-ssh-` — deliberately failing the remote Codeman's `SAFE_MUX_NAME_PATTERN` so a Codeman instance on the target host never adopts it; no `-g` global tmux options are set remotely. `remotePath`/`identityFile` are schema-guarded against shell injection (backticks/`$` rejected — same approach as `extraSshOptions`); remote tmux availability is probed via `checkRemoteTmuxAvailable()` in quick-start (ssh args carry `-o ConnectTimeout=10`). Remote claude defaults to `exec claude --dangerously-skip-permissions`; per-host `commands.*` override. Session kill best-effort kills the remote tmux too. `SessionState.remote`/`MuxSession.remote` round-trip through recovery (`restoreMuxSessions` passes `remote` back into the Session constructor). ⚠️ Run flows must route remote cases through `POST /api/quick-start` (which resolves the remote case and skips LOCAL CLI availability gates) — `POST /api/sessions` stat-validates `workingDir` locally and has no `caseName`. `envOverrides`/`effort`/`modelOverride`/`codexConfig`/`geminiConfig` are rejected for remote quick-starts (not silently dropped). UI: Create Case modal → Remote tab. Tests: `test/remote-hosts.test.ts`, `test/remote-ssh-options.test.ts`. diff --git a/docs/remote-sessions.md b/docs/remote-sessions.md new file mode 100644 index 00000000..5c1f9cb2 --- /dev/null +++ b/docs/remote-sessions.md @@ -0,0 +1,237 @@ +# Remote Sessions (SSH) + +Codeman can run a session's agent on a **remote host over SSH** instead of the +local machine. The agent (Claude, OpenCode, Codex, Gemini, or a plain shell) +runs inside a `tmux` server **on the remote host**, so it survives the SSH +connection dropping; Codeman attaches to it the same way it attaches to a local +managed session. + +This document covers the data model, the shell-safe SSH command construction +(COD-107), the durable-launch design (COD-104), and the operational caveats. +For the local session/mux machinery this builds on, see the **Mux** and +**Session** entries in `CLAUDE.md` → Architecture. + +## Why it exists + +A developer box (`AA-DESKTOP`) often needs to drive an agent on another machine — +a NAS, a build server, a host reachable only through a jump box or a +cloudflared SOCKS5 proxy. Rather than wrap `ssh` by hand per host, Codeman +stores reusable **remote hosts** + **remote cases** and reproduces the exact +connection the operator already uses (`ssh-aa-desktop`-style configs: +custom port, identity file, `-J` jump host, `-o ProxyCommand`). + +## Data model + +Types live in `src/types/session.ts`; persistence in `src/remote-hosts.ts`. + +| Type | Role | +|------|------| +| `RemoteSshOptions` | The **HOW-to-reach** fields, shared by host + session: `identityFile`, `socksProxy` (`host:port`), `jumpHost` (`[user@]host[:port]`), `extraSshOptions` (`KEY=VALUE[]`). Every field optional — all-absent reproduces port-22, default-identity, directly-SSH-able behavior. | +| `RemoteHost` (extends `RemoteSshOptions`) | A saved host: `id`, `label`, `host`, `username`, `port?`, `commands?` (per-mode launch command override). | +| `RemoteCase` | A working directory on a host: `name`, `type: 'remote'`, `hostId`, `remotePath`. | +| `SessionRemote` (extends `RemoteSshOptions`) | The resolved bundle stamped onto a live session: host coordinates + `remotePath` + `commands`, plus **`owned?`** and **`remoteSessionName?`** (COD-105 — see [Ownership](#ownership-launched-vs-discovered-and-attached-cod-105)). Built by `toSessionRemote(host, case)` (sets `owned: true`) for the launch path, or `toAttachedSessionRemote(host, name, path)` (sets `owned: false`) for the attach path. Both copy the advanced SSH options through so every connection is identical. | +| `RemoteCommandMode` | `Extract` — the modes that can run remotely. | +| `RemoteSessionInfo` (COD-105) | One discovered remote tmux session: `name` (always `codeman-*`), `attached` (a client is connected), `created` (epoch s), `windows`. Returned by `listRemoteCodemanSessions()`. | + +Persistence is two flat JSON arrays in the instance data dir: + +- `~/.codeman/remote-hosts.json` — `readRemoteHosts()` / `writeRemoteHosts()` +- `~/.codeman/remote-cases.json` — `readRemoteCases()` / `writeRemoteCases()` + +(Paths via `remoteHostsPath()` / `remoteCasesPath()`; both honor `CODEMAN_INSTANCE` +because the config dir is the instance data dir.) + +On the live `Session`, the remote rides as `_remote?: SessionRemote`. When +attaching, `resolveMuxAttachCwd()` forces the cwd to `/tmp` for remote sessions — +the local working directory is meaningless on the remote box. + +## SSH command construction (COD-107 — the injection surface) + +**All** SSH command lines flow through one function so user-controlled fields are +escaped once and the launch + prereq probe can never drift apart: + +```ts +// src/remote-hosts.ts +buildSshConnectionArgs(remote: RemoteSshOptions & Pick): string[] +``` + +It returns the **ordered leading tokens** of an ssh command line (no `-t`, no +target, no remote command): + +``` +ssh -o BatchMode=yes + [-p ] + [-i ] # ~ / $HOME expanded, then shellescaped + [-J ] # shellescaped, single token + [-o ProxyCommand=nc -X 5 -x %h %p] # ONE shellescaped -o token + [-o ] … # each extra option, shellescaped +``` + +Rules that keep this safe — **do not bypass them by hand-building an ssh line elsewhere:** + +- **Every** user-controlled value (`-i`, `-J`, `-o`, ProxyCommand) is POSIX + single-quote `shellescape`d (`'…'` with embedded `'\''`). The helper mirrors + the one in `tmux-manager.ts`. +- **`~`/`$HOME` in `identityFile` is expanded at build time** (`expandIdentityPath`), + *before* escaping — ssh does not expand `~` inside `-i`, and the escaped value + never reaches a shell that would. +- **The ProxyCommand is one shellescaped `-o KEY=VALUE` token**, so its spaces and + the `%h`/`%p` placeholders reach ssh as a single argument. `%h %p` survive + verbatim — **ssh** expands them to the real host/port, not the shell. +- **Empty options ⇒ `['ssh', '-o BatchMode=yes']`** (+ `-p` only when set) — + byte-identical to the historical behavior. + +Token construction is unit-tested independently of any live connection (see +`test/` for `buildSshConnectionArgs` / `buildRemoteTmuxCheckCommand` cases). + +## Durable launch (COD-104) + +`buildRemoteLaunchCommand({ mode, remote, sessionId })` in `tmux-manager.ts` +builds the command that launches (or **reattaches** to) the remote session: + +``` +ssh -o BatchMode=yes -t user@host \ + 'tmux -L codeman new-session -A -s codeman- -c "cd && exec " \; \ + set -g status off \; set -g mouse off \; set -sg escape-time 0 \; set -g prefix C-q' +``` + +Key points: + +- **`new-session -A -s codeman-`** = attach-if-exists-else-create, so a + reconnect (same deterministic `remoteTmuxSessionName(sessionId)`) lands back in + the **same** remote session rather than spawning a duplicate. This is what makes + the remote agent survive an SSH drop. +- **`-L codeman`** = canonical remote socket (the remote's own tmux server, not + the local one). +- **`exec `** replaces the pane shell with the agent, so the pane PID *is* + the agent. The per-mode command comes from `remote.commands?.[mode]` or + `defaultRemoteCommandForMode(mode)` (`exec claude` / `exec opencode` / + `exec codex` / `exec gemini` / `exec bash -l`). +- The **whole tmux invocation is a single shell-quoted ssh argument**, and the + pane command is independently quoted, so a `remotePath` with spaces is safe. +- Connection options come from the **same `buildSshConnectionArgs(remote)`** as + the prereq probe; `-t` is inserted right after `ssh -o BatchMode=yes`, + preserving historical token order. + +### tmux prerequisite probe + +Because durable remote sessions require tmux on the remote host, +`checkRemoteTmuxAvailable(host)` runs `command -v tmux` over SSH **before** +creating a remote case/session and returns a structured, never-throwing result: + +- empty stdout / non-zero exit → *"remote host `` needs tmux installed for + durable remote sessions"* +- stderr present → *"could not verify tmux on remote host ``: ``"* + (a real connection failure, surfaced to the operator) +- success → `{ ok: true, tmuxPath }` + +It connects with the **identical** options as the launch +(`buildRemoteTmuxCheckCommand` reuses `buildSshConnectionArgs` and inserts +`-o ConnectTimeout=10`), so a proxied/custom-port/identity host that the launch +can reach also passes the probe (and vice-versa). + +**Test-mode short-circuit:** under `VITEST` the probe returns +`{ ok: true, tmuxPath: '(test-mode)' }` without opening a socket — mirroring +`TmuxManager`'s no-op-shell-under-VITEST (`IS_TEST_MODE`). Without it, remote-case +create-path tests would hit a real ~10s ssh timeout. Only the live probe is +skipped; command construction is still asserted by unit tests. + +## Ownership: launched vs. discovered-and-attached (COD-105) + +COD-104 (above) was Phase 1 — Codeman *launches* a remote session and owns it. +COD-105 is Phase 2 — Codeman can also **discover** `codeman-*` tmux sessions +already running on a remote host (created by the remote's own Codeman or another +instance) and **attach** to one it didn't launch. Ownership decides what happens +when the tab closes. + +`SessionRemote.owned` carries this: + +- **`owned: true`** (or absent — legacy/COD-104 sessions persisted before this + field) — we launched it via `buildRemoteLaunchCommand` and may explicitly kill it. +- **`owned: false`** — discovered + attached; another Codeman owns the remote + session. `remoteSessionName` holds its existing tmux name. Closing the tab + **detaches**, never kills. + +### Discovery + +`listRemoteCodemanSessions(host)` lists the remote's `codeman-*` sessions: + +- `buildRemoteListSessionsCommand()` runs `tmux -L codeman list-sessions -F "…"` + over SSH (connection args from the shared `buildSshConnectionArgs`, so discovery + connects identically to launch/probe). `2>/dev/null` swallows tmux's "no server + running" stderr. +- `parseRemoteSessionList()` is a **pure, unit-tested** parser. ⚠️ Quirk: the + remote tmux's `-F "…\t…"` format emits the **literal two-character `\t`**, not a + real tab (verified on tmux next-3.7), so the parser splits on `/\\t|\t/` (literal + backslash-t **or** a real tab, for builds that do expand it). It keeps only + `codeman-*` names, coerces types, and skips malformed lines. +- `listRemoteCodemanSessions()` **never throws** — unreachable host / no tmux / no + sessions all map to `[]`. Like the prereq probe, it **no-ops to `[]` under + `VITEST`** so a request path never opens a real ssh connection. + +Discovery is **explicit** — the UI has a "Discover existing sessions" button per +host; Codeman never auto-discovers on host select. + +### Attach vs. launch selection + +`buildRemoteSessionCommand(mode, remote, sessionId)` in `tmux-manager.ts` picks the +remote command line by ownership: + +- **`owned === false`** → `buildRemoteAttachCommand(remote, name)` — emits + `ssh … -t … 'tmux -L codeman attach -t '`. It uses **`attach`, + NOT `new-session -A`**, so it only *joins* an existing session and never creates + one. +- **owned (default)** → `buildRemoteLaunchCommand` (the COD-104 path above). + +### Detach-not-kill + +`TmuxManager.killSession()` has an **early return for non-owned remote sessions**: +it tears down **only the LOCAL pane** holding the ssh client (`tmux -L codeman +kill-session` on *this* host's socket). Killing the local ssh sends SIGHUP to the +remote `tmux attach`, which **detaches** — the durable remote session survives. +The early return is a structural guarantee that **no code path can ever issue a +remote `kill-session` for a session we don't own** — the only `kill-session` run is +on the local socket, which never reaches the remote socket. + +## API + +Routes are registered in `src/web/routes/case-routes.ts`: + +| Method | Path | Purpose | +|--------|------|---------| +| `GET` | `/api/remote-hosts` | List saved hosts | +| `POST` | `/api/remote-hosts` | Create a host | +| `PUT` | `/api/remote-hosts/:id` | Update a host | +| `DELETE` | `/api/remote-hosts/:id` | Delete a host | +| `GET` | `/api/remote-hosts/:hostId/sessions` | Discover `codeman-*` sessions on the host (COD-105; `listRemoteCodemanSessions`, never errors) | +| `POST` | `/api/cases/remote-link` | Link a case to a remote host (creates the `RemoteCase`) | + +Attaching to a discovered session is a **session-create** path, not a host route: +`POST /api/sessions` accepts `attachRemoteSession: { hostId, remoteSessionName }` +(schema in `schemas.ts`; `remoteSessionName` must match `^codeman-[a-zA-Z0-9._-]+$`), +which `session-routes.ts` turns into a non-owned (`owned: false`) session. + +Frontend touchpoints: the remote-host management UI is in `session-ui.js` / +`panels-ui.js`; a remote session is created by picking a remote host/case in the +session-create flow, or via the per-host **"Discover existing sessions"** button → +**Attach** action (creates an `owned: false` session). + +## Security notes + +- **`identityFile` is a path only — never key bytes.** Codeman stores the path and + passes it to `ssh -i`; the key never enters Codeman's state or the wire. +- The injection surface is the SSH option fields. The single-source + `buildSshConnectionArgs` + `shellescape` discipline (COD-107) is the control — + audit any new code path that constructs an ssh command to route through it + rather than concatenating options inline. +- `BatchMode=yes` means **no interactive password/passphrase prompts** — remote + hosts must be reachable with key-based or agent auth (or an unencrypted key the + agent has loaded). A host needing a passphrase will fail the probe with an ssh + diagnostic rather than hang. + +## Related + +- `CLAUDE.md` → Architecture → **Remote** row, and the **Remote sessions (SSH)** + Key Pattern. +- `docs/security-architecture.md` — overall network/auth model. +- COD-104 (tmux prereq + durable launch), COD-105 (discover + attach, detach-not-kill ownership), COD-107 (shell-safe connection args). diff --git a/src/remote-hosts.ts b/src/remote-hosts.ts index dbf82cff..571924a3 100644 --- a/src/remote-hosts.ts +++ b/src/remote-hosts.ts @@ -8,6 +8,7 @@ import type { RemoteCase, RemoteCommandMode, RemoteHost, + RemoteSessionInfo, RemoteSshOptions, SessionMode, SessionRemote, @@ -173,6 +174,14 @@ export interface RemoteTmuxCheckResult { export async function checkRemoteTmuxAvailable( host: Pick & RemoteSshOptions ): Promise { + // Under vitest, never open a real ssh connection — mirrors TmuxManager's + // no-op-shell-under-VITEST (IS_TEST_MODE). Without this, remote-case + // create-path tests hit a real ~10s ssh timeout. The command construction is + // covered by buildRemoteTmuxCheckCommand unit tests; only the live probe is + // short-circuited here. + if (process.env.VITEST) { + return { ok: true, tmuxPath: '(test-mode)' }; + } const command = buildRemoteTmuxCheckCommand(host); try { const { stdout } = await execAsync(command, { timeout: 15_000 }); @@ -202,6 +211,109 @@ export async function checkRemoteTmuxAvailable( } } +/** + * COD-105 — build the SSH command that lists `codeman-*` tmux sessions on a + * remote host's canonical `-L codeman` socket. + * + * `list-sessions` exits NON-ZERO with empty output when no sessions exist (and + * the server isn't running), so `2>/dev/null` swallows tmux's "no server + * running" stderr; the caller treats a non-zero exit / empty output as "no + * sessions" rather than an error. + * + * COD-107 — connection options come from the shared `buildSshConnectionArgs`, so + * discovery connects with the SAME port/identity/proxy/jump-host as the launch + * and the tmux prereq probe. + */ +export function buildRemoteListSessionsCommand( + host: Pick & RemoteSshOptions +): string { + const [ssh, ...connectionArgs] = buildSshConnectionArgs(host); + const parts = [ssh, connectionArgs[0], '-o ConnectTimeout=10', ...connectionArgs.slice(1)]; + // The tmux list-sessions invocation is passed as ONE shell-quoted argument so + // the remote login shell runs it verbatim. The `-F` format uses literal `\t` + // separators (tmux expands them); `2>/dev/null` is inside the quoted command. + const remoteCmd = + 'tmux -L codeman list-sessions -F "#{session_name}\\t#{session_attached}\\t#{session_created}\\t#{session_windows}" 2>/dev/null'; + parts.push(remoteSshTarget(host), shellescape(remoteCmd)); + return parts.join(' '); +} + +/** + * COD-105 — pure parser for the `tmux list-sessions -F` output emitted by + * `buildRemoteListSessionsCommand`. Factored out so the parse is unit-testable + * without opening a real ssh connection. + * + * - Splits each non-empty line into [name, attached, created, windows] on the + * field separator. IMPORTANT: the remote tmux's `-F "…\t…"` format does NOT + * expand `\t` to a real tab — it emits the LITERAL two-character sequence + * `\t` (verified on aa-desktop / tmux next-3.7). So we split on the literal + * backslash-t sequence; we also tolerate a real tab in case a tmux build + * does expand it. (A real TAB is the regex `\t`; a literal backslash-t is the + * regex `\\t`.) + * - Keeps ONLY sessions whose name starts with `codeman-` (ignores foreign tmux + * sessions that happen to share the socket). + * - Coerces: `attached` → boolean (`'1'`), `created`/`windows` → finite ints. + * - Skips malformed lines (wrong column count or non-numeric created/windows) + * rather than emitting garbage. + */ +export function parseRemoteSessionList(stdout: string): RemoteSessionInfo[] { + const out: RemoteSessionInfo[] = []; + for (const rawLine of stdout.split('\n')) { + const line = rawLine.trim(); + if (!line) continue; + // Split on a literal `\t` (backslash + t, what the remote tmux emits) OR a + // real tab character. `/\\t|\t/` = the two-char sequence, or a TAB. + const cols = line.split(/\\t|\t/); + if (cols.length !== 4) continue; + const [name, attachedStr, createdStr, windowsStr] = cols; + if (!name.startsWith('codeman-')) continue; + const created = Number(createdStr); + const windows = Number(windowsStr); + if (!Number.isFinite(created) || !Number.isFinite(windows)) continue; + // COD-106 — `session_attached` is the CLIENT COUNT (not a 0/1 flag); >1 = shared. + const attachedNum = Number(attachedStr.trim()); + const attachedClients = Number.isFinite(attachedNum) ? Math.max(0, Math.trunc(attachedNum)) : 0; + out.push({ + name, + attached: attachedClients > 0, + attachedClients, + created: Math.trunc(created), + windows: Math.trunc(windows), + }); + } + return out; +} + +/** + * COD-105 — discover `codeman-*` tmux sessions already running on a remote host + * (created by the remote's own Codeman, another instance, or this one), so the + * operator can attach to one this Codeman didn't launch. + * + * NEVER throws: returns `[]` on unreachable host / no tmux / no sessions + * (`list-sessions` exits non-zero with empty output when there are none). + * + * VITEST guard — like `checkRemoteTmuxAvailable`, returns `[]` under test so a + * real ssh never runs in a request path (which would make route tests hit a + * ~10s timeout). The command construction is covered by + * `buildRemoteListSessionsCommand` and the parse by `parseRemoteSessionList`. + */ +export async function listRemoteCodemanSessions( + remote: Pick & RemoteSshOptions +): Promise { + if (process.env.VITEST) { + return []; + } + const command = buildRemoteListSessionsCommand(remote); + try { + const { stdout } = await execAsync(command, { timeout: 15_000 }); + return parseRemoteSessionList(stdout); + } catch { + // Unreachable host, no tmux server, or no sessions (non-zero exit). All map + // to "nothing to attach to" — never surface as an error to the caller. + return []; + } +} + export function remoteDisplayPath( remote: Pick | { username: string; host: string; path: string } ): string { @@ -218,6 +330,10 @@ export function toSessionRemote(host: RemoteHost, remoteCase: RemoteCase): Sessi port: host.port, remotePath: remoteCase.remotePath, commands: host.commands, + // COD-105 — the COD-104 launch path creates the remote session, so we own it + // (an explicit kill may propagate a remote kill-session). Discovered+attached + // sessions go through `toAttachedSessionRemote` with `owned: false`. + owned: true, // COD-107 — carry the advanced SSH options from host config into the session // so the launch/prereq commands connect the same way the operator configured. identityFile: host.identityFile, @@ -226,3 +342,38 @@ export function toSessionRemote(host: RemoteHost, remoteCase: RemoteCase): Sessi extraSshOptions: host.extraSshOptions, }; } + +/** + * COD-105 — build a NON-owned `SessionRemote` for ATTACHING to a `codeman-*` + * session already running on a remote host (discovered via + * `listRemoteCodemanSessions`). The resulting session's pane runs + * `tmux -L codeman attach -t ` (see + * `buildRemoteAttachCommand`), and because we did NOT create the remote session, + * `owned: false` means closing the tab DETACHES rather than killing it. + * + * `remotePath` is informational here (the attached remote session keeps its own + * cwd); we record the host's nominal path so display helpers still show + * `user@host:path`. + */ +export function toAttachedSessionRemote( + host: RemoteHost, + remoteSessionName: string, + remotePath: string +): SessionRemote { + return { + hostId: host.id, + label: host.label, + host: host.host, + username: host.username, + port: host.port, + remotePath, + commands: host.commands, + // Discovered + attached — another Codeman created it. Detach-not-kill. + owned: false, + remoteSessionName, + identityFile: host.identityFile, + socksProxy: host.socksProxy, + jumpHost: host.jumpHost, + extraSshOptions: host.extraSshOptions, + }; +} diff --git a/src/remote-reconnect.ts b/src/remote-reconnect.ts new file mode 100644 index 00000000..6e5a709a --- /dev/null +++ b/src/remote-reconnect.ts @@ -0,0 +1,184 @@ +/** + * @fileoverview Pure logic for the remote-session auto-reconnect watcher (COD-108). + * + * COD-104 made remote tmux sessions durable + idempotently reattachable, but a + * reconnect only fired at explicit trigger points. COD-108 adds a continuous + * watcher (in `TmuxManager`) that detects a dead remote pane and emits + * `remoteSessionDropped`; `SessionManager`/server then reassembles the respawn + * options and reattaches (re-running the idempotent remote command). + * + * This module holds the SIDE-EFFECT-FREE pieces so they can be unit-tested + * without real tmux: + * - the bounded exponential **backoff schedule** (attempt → delay, capped), + * - the per-session **reconnect state** shape, + * - the **eligibility decision** (`decideReconnect`) given a session + its + * reconnect state + the current time + the guard set. + * + * The watcher in `tmux-manager.ts` owns the live `isPaneDead` probe and the + * timers; everything here is pure and deterministic (time is injected). + * + * @module remote-reconnect + */ + +/** + * Bounded exponential backoff delays (ms) between reconnect attempts. + * Attempt N (1-based) waits `BACKOFF_SCHEDULE_MS[N-1]` from the previous emit + * before the next emit is eligible. After the last entry the session is + * considered `reconnect-exhausted` and the watcher stops emitting for it. + * + * 5s, 15s, 45s, 2m, 5m, 5m → ~6 attempts spanning ~13 minutes. + */ +export const BACKOFF_SCHEDULE_MS: readonly number[] = [5_000, 15_000, 45_000, 120_000, 300_000, 300_000]; + +/** Maximum number of reconnect attempts before exhaustion. */ +export const MAX_RECONNECT_ATTEMPTS = BACKOFF_SCHEDULE_MS.length; + +/** + * Delay (ms) to wait AFTER emitting attempt `attempt` (1-based) before the next + * attempt is eligible. `attempt <= 0` returns the first delay; an attempt at or + * beyond the cap returns the last delay (callers should check exhaustion via + * {@link isExhausted} rather than relying on this for the stop decision). + * + * Pure — no clock, no I/O. + */ +export function reconnectDelayForAttempt(attempt: number): number { + if (!Number.isFinite(attempt) || attempt <= 1) return BACKOFF_SCHEDULE_MS[0]; + const idx = Math.min(Math.floor(attempt) - 1, BACKOFF_SCHEDULE_MS.length - 1); + return BACKOFF_SCHEDULE_MS[idx]; +} + +/** Whether `attempts` reconnect emits have reached/exceeded the cap. Pure. */ +export function isExhausted(attempts: number): boolean { + return attempts >= MAX_RECONNECT_ATTEMPTS; +} + +/** + * Per-session reconnect bookkeeping held by the watcher. All time values are + * epoch ms. `inFlight` guards against stacking respawns when a tick fires while + * a previous reattach is still running. `exhaustedEmitted` ensures the + * `remoteReconnectExhausted` event fires at most once per session. + */ +export interface RemoteReconnectState { + /** Number of `remoteSessionDropped` emits so far (advances per emit). */ + attempts: number; + /** Earliest time (epoch ms) the next emit is eligible. 0 = eligible now. */ + nextEligibleAt: number; + /** A reattach triggered by a prior emit is currently running. */ + inFlight: boolean; + /** Cap reached — stop auto-retrying for this session. */ + exhausted: boolean; + /** The `remoteReconnectExhausted` SSE event has already been emitted. */ + exhaustedEmitted: boolean; +} + +/** A fresh reconnect state (no attempts, immediately eligible). Pure. */ +export function freshReconnectState(): RemoteReconnectState { + return { attempts: 0, nextEligibleAt: 0, inFlight: false, exhausted: false, exhaustedEmitted: false }; +} + +/** + * Advance the backoff after an emit at time `now`. Increments `attempts` and + * schedules `nextEligibleAt = now + delay`. Returns a NEW state object (does + * not mutate the input). Pure. + * + * NOTE: this does NOT set `exhausted`. Exhaustion is a decision the watcher + * makes on the FOLLOWING tick (via {@link decideReconnect} → `exhaust`), so the + * `remoteReconnectExhausted` event fires exactly once after the final attempt's + * backoff window elapses — not pre-emptively on the last emit. + */ +export function advanceBackoff(state: RemoteReconnectState, now: number): RemoteReconnectState { + const attempts = state.attempts + 1; + const delay = reconnectDelayForAttempt(attempts); + return { + ...state, + attempts, + nextEligibleAt: now + delay, + }; +} + +/** Reset after a successful reattach — back to a fresh, eligible state. Pure. */ +export function resetReconnectState(): RemoteReconnectState { + return freshReconnectState(); +} + +/** Minimal session view the decision needs (avoids importing MuxSession here). */ +export interface ReconnectSessionView { + sessionId: string; + /** Truthy when this is a remote (SSH-wrapped) session. */ + isRemote: boolean; + /** Result of `isPaneDead(muxName)` for this session. */ + paneDead: boolean; +} + +/** + * Decision outcomes for a single watcher tick on one session. + * - `emit` → emit `remoteSessionDropped { sessionId, attempt }`, then + * advance backoff (attempt = the returned `attempt`). + * - `exhaust` → cap reached this tick; emit `remoteReconnectExhausted` once. + * - `skip` → do nothing (not remote / pane alive / guarded / in-flight / + * not yet due / already exhausted). + */ +export type ReconnectAction = + | { kind: 'emit'; attempt: number } + | { kind: 'exhaust' } + | { kind: 'skip'; reason: ReconnectSkipReason }; + +export type ReconnectSkipReason = + | 'not-remote' + | 'pane-alive' + | 'guarded' + | 'in-flight' + | 'not-due' + | 'exhausted' + | 'disabled'; + +export interface DecideReconnectInput { + session: ReconnectSessionView; + state: RemoteReconnectState | undefined; + /** Session is in the intentional-teardown guard set (killed/detached/stopping). */ + guarded: boolean; + /** Kill-switch: `remoteAutoReconnect` setting. When false, never reconnect. */ + enabled: boolean; + now: number; +} + +/** + * PURE eligibility decision for one session on one tick. No clock, no I/O — all + * inputs are passed in. The watcher translates the result into emits + state + * transitions. + * + * Order of guards (most-decisive first): + * 1. kill-switch off → skip:disabled + * 2. not a remote session → skip:not-remote + * 3. pane is alive → skip:pane-alive + * 4. intentional teardown guard → skip:guarded (NEVER revive a killed tab) + * 5. a reattach already running → skip:in-flight (no stacked respawns) + * 6. already exhausted → skip:exhausted (one exhaust emit, then quiet) + * 7. cap reached this tick → exhaust + * 8. not yet due (backoff) → skip:not-due + * 9. otherwise → emit (attempt = attempts + 1) + */ +export function decideReconnect(input: DecideReconnectInput): ReconnectAction { + const { session, state, guarded, enabled, now } = input; + + if (!enabled) return { kind: 'skip', reason: 'disabled' }; + if (!session.isRemote) return { kind: 'skip', reason: 'not-remote' }; + if (!session.paneDead) return { kind: 'skip', reason: 'pane-alive' }; + // Intentional kill / detach must NEVER be auto-revived. + if (guarded) return { kind: 'skip', reason: 'guarded' }; + + const s = state ?? freshReconnectState(); + + // Only one reconnect in flight per session — don't stack respawns. + if (s.inFlight) return { kind: 'skip', reason: 'in-flight' }; + + if (s.exhausted) return { kind: 'skip', reason: 'exhausted' }; + + // Cap reached: surface exhaustion once, then go quiet. + if (isExhausted(s.attempts)) return { kind: 'exhaust' }; + + // Backoff gate — only emit when due. + if (now < s.nextEligibleAt) return { kind: 'skip', reason: 'not-due' }; + + return { kind: 'emit', attempt: s.attempts + 1 }; +} diff --git a/src/session.ts b/src/session.ts index a509040a..3f344fc9 100644 --- a/src/session.ts +++ b/src/session.ts @@ -1211,6 +1211,68 @@ export class Session extends EventEmitter { return { isRestored }; } + /** + * COD-108 — re-establish a dropped REMOTE session. Triggered by the + * `TmuxManager` remote-reconnect watcher (via `remoteSessionDropped`): the + * watcher detects a dead remote pane, the session owner reassembles the SAME + * `RespawnPaneOptions` used for Claude-idle respawns and calls + * `respawnPane()` directly. For a remote session that re-runs + * `buildRemoteSessionCommand` (owned → `new-session -A`, non-owned → + * `attach`), which idempotently REATTACHES the still-running durable remote + * tmux session — scrollback + agent intact (proven COD-104/105). + * + * Deliberately does NOT route through the Claude-idle respawn-controller — + * this is a transport re-establish, not a `/clear`/`/compact` cycle. + * + * @returns true if the pane was respawned (reattach issued), false otherwise. + */ + async reattachRemote(): Promise { + if (!this._remote) return false; // not a remote session + if (!this._useMux || !this._mux || !this._muxSession) return false; + const mux = this._mux; + + // If tmux lost the whole session (not just a dead pane), there is nothing to + // respawn into — a genuine death, leave it for normal recovery/reconcile. + if (!mux.muxSessionExists(this._muxSession.muxName)) { + console.log('[Session] reattachRemote: mux session gone, skipping:', this._muxSession.muxName); + return false; + } + + const newPid = await mux.respawnPane(this._buildRespawnPaneOptions()); + if (!newPid) { + console.error('[Session] reattachRemote: respawnPane failed for', this._muxSession.muxName); + return false; + } + console.log('[Session] reattachRemote: reattached remote session', this._muxSession.muxName, 'pid', newPid); + return true; + } + + /** + * Assemble the {@link RespawnPaneOptions} for this session. Single source of + * truth shared by interactive start, shell start (via their inline copies), + * and {@link reattachRemote} so the remote reattach path can never drift from + * the spawn path. + */ + private _buildRespawnPaneOptions(): import('./mux-interface.js').RespawnPaneOptions { + return { + sessionId: this.id, + workingDir: this.workingDir, + mode: this.mode, + niceConfig: this._niceConfig, + model: this._model, + claudeMode: this._claudeMode, + allowedTools: this._allowedTools, + openCodeConfig: this._openCodeConfig, + codexConfig: this._codexConfig, + geminiConfig: this._geminiConfig, + resumeSessionId: this._resumeSessionId, + envOverrides: this._envOverrides, + effort: this._effort, + historyLimit: this._tmuxHistoryLimit, + remote: this._remote, + }; + } + private _handleTerminalOutput(data: string): void { // Codex AND Claude Code emit sequences that wipe xterm.js scrollback, plus // mouse-tracking enables that hijack the scroll wheel so the user can't reach @@ -1334,23 +1396,8 @@ export class Session extends EventEmitter { if (this._useMux && this._mux) { try { const { isRestored } = await this._setupOrAttachMuxSession({ - respawnPaneOptions: { - sessionId: this.id, - workingDir: this.workingDir, - mode: this.mode, - niceConfig: this._niceConfig, - model: this._model, - claudeMode: this._claudeMode, - allowedTools: this._allowedTools, - openCodeConfig: this._openCodeConfig, - codexConfig: this._codexConfig, - geminiConfig: this._geminiConfig, - resumeSessionId: this._resumeSessionId, - envOverrides: this._envOverrides, - effort: this._effort, - historyLimit: this._tmuxHistoryLimit, - remote: this._remote, - }, + // Single source of truth shared with reattachRemote() (COD-108). + respawnPaneOptions: this._buildRespawnPaneOptions(), createSessionOptions: { sessionId: this.id, workingDir: this.workingDir, diff --git a/src/tmux-manager.ts b/src/tmux-manager.ts index 0139d20d..24b5d539 100644 --- a/src/tmux-manager.ts +++ b/src/tmux-manager.ts @@ -62,6 +62,13 @@ import type { RespawnPaneOptions, PaneCaptureOptions, } from './mux-interface.js'; +import { + decideReconnect, + advanceBackoff, + freshReconnectState, + resetReconnectState, + type RemoteReconnectState, +} from './remote-reconnect.js'; // ============================================================================ // Timing Constants @@ -94,6 +101,9 @@ const GRACEFUL_SHUTDOWN_WAIT_MS = 100; /** Default stats collection interval (2 seconds) */ const DEFAULT_STATS_INTERVAL_MS = 2000; +/** Default remote-reconnect watcher poll interval (5 seconds) — COD-108 */ +const DEFAULT_REMOTE_RECONNECT_INTERVAL_MS = 5000; + /** Stable cwd for tmux server/pane launch; actual session cwd is reached inside the pane. */ const TMUX_LAUNCH_CWD = '/tmp'; @@ -118,6 +128,20 @@ const IS_TEST_MODE = !!process.env.VITEST; /** Path to persisted mux session metadata */ const MUX_SESSIONS_FILE = dataPath('mux-sessions.json'); +/** + * COD-108 kill-switch: `remoteAutoReconnect` app setting (default ON). Read at + * call time (like headroom routing) so a settings change takes effect without a + * restart. Absent/non-boolean ⇒ true (feature on). + */ +function isRemoteAutoReconnectEnabled(): boolean { + try { + const s = JSON.parse(readFileSync(dataPath('settings.json'), 'utf8')) as Record; + return typeof s.remoteAutoReconnect === 'boolean' ? s.remoteAutoReconnect : true; + } catch { + return true; + } +} + /** Regex to validate tmux session names (only allow safe characters) */ const SAFE_MUX_NAME_PATTERN = /^codeman-[a-f0-9-]+$/; @@ -781,6 +805,13 @@ export function buildRemoteLaunchCommand(options: { `set -t ${remoteName} mouse off`, `set -t ${remoteName} prefix C-q`, 'set -s escape-time 0', + // COD-106 — shared/collaborative sessions: tmux defaults to sizing a window + // to the SMALLEST attached client, so two Codemans at different viewports + // would fight (clamp to the smaller). `window-size latest` sizes to the + // most-recently-active client instead, so concurrent clients coexist. + // Per-session scoped (`set -t `, matching #145's hardening) so a shared + // remote tmux server's other sessions keep their own sizing behavior. + `set -t ${remoteName} window-size latest`, ].join(' \\; '); // ssh runs its trailing args through the remote login shell, so the entire @@ -812,6 +843,49 @@ export function buildRemoteKillCommand(options: { remote: SessionRemote; session return [ssh, ...connectionArgs, remoteSshTarget(remote), shellescape(killCmd)].join(' '); } +/** + * COD-105 — build the SSH command that ATTACHES to an EXISTING `codeman-*` tmux + * session on the remote host (one this Codeman didn't create — discovered via + * `listRemoteCodemanSessions`). Sibling of `buildRemoteLaunchCommand`. + * + * Emits: + * ssh -o BatchMode=yes -t [] user@host \ + * 'tmux -L codeman attach -t ' + * + * - `attach` (NOT `new-session -A`) so we only join an existing session; the + * remote session keeps running independent of us, which is exactly why the + * resulting Codeman session is NON-OWNED (see `SessionRemote.owned`): closing + * the local tab must detach, never `kill-session` the remote. + * - The remote session name is shell-escaped so a value with metachars stays a + * single token inside the quoted tmux invocation. + * - COD-107 — connection options (`-p`, `-i`, `-J`, SOCKS `-o ProxyCommand`, + * arbitrary `-o`) come from the shared `buildSshConnectionArgs`, so attach + * connects identically to launch / discovery / the prereq probe. `-t` sits + * right after `ssh -o BatchMode=yes` (a PTY is required for interactive tmux). + */ +export function buildRemoteAttachCommand(remote: SessionRemote, remoteSessionName: string): string { + const tmuxInvocation = `tmux -L codeman attach -t ${shellescape(remoteSessionName)}`; + const [ssh, batchMode, ...connectionArgs] = buildSshConnectionArgs(remote); + const sshParts = [ssh, batchMode, '-t', ...connectionArgs, remoteSshTarget(remote), shellescape(tmuxInvocation)]; + return sshParts.join(' '); +} + +/** + * COD-105 — choose the right remote ssh command for a session's ownership: + * - NON-owned (`remote.owned === false`): ATTACH to a discovered remote tmux + * session by its EXISTING name (`remote.remoteSessionName`, falling back to + * this session's deterministic name). We only join — never create. + * - owned (default): LAUNCH/attach-or-create via `buildRemoteLaunchCommand` + * (COD-104), which we then own and may explicitly kill. + */ +function buildRemoteSessionCommand(mode: SessionMode, remote: SessionRemote, sessionId: string): string { + if (remote.owned === false) { + const target = remote.remoteSessionName || remoteTmuxSessionName(sessionId); + return buildRemoteAttachCommand(remote, target); + } + return buildRemoteLaunchCommand({ mode, remote, sessionId }); +} + /** * Set sensitive environment variables on a tmux session via setenv. * These are inherited by panes but not visible in ps output or tmux history. @@ -965,6 +1039,17 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { /** Track last-known pane count per session to avoid unnecessary tmux set-option calls */ private lastPaneCount: Map = new Map(); + // ── COD-108 remote-reconnect watcher state ──────────────────────────────── + /** Periodic watcher that re-establishes dropped remote sessions. */ + private remoteReconnectInterval: NodeJS.Timeout | null = null; + /** Per-session backoff/attempt bookkeeping (sessionId → state). */ + private reconnectState: Map = new Map(); + /** + * Sessions excluded from auto-reconnect because they are being intentionally + * torn down (killed/detached/stopping). A guarded session is NEVER revived. + */ + private reconnectGuard: Set = new Set(); + private trueColorConfigured = false; constructor() { @@ -1273,7 +1358,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { try { // Build the full command to run inside tmux const localFullCmd = `${buildNofileLimitCommand()} && ${pathExport}${envExportsStr} && ${cmd}`; - const fullCmd = remote ? buildRemoteLaunchCommand({ mode, remote, sessionId }) : localFullCmd; + const fullCmd = remote ? buildRemoteSessionCommand(mode, remote, sessionId) : localFullCmd; // Create tmux session in three steps to handle cold-start (no server running) // and avoid the race where the command exits before remain-on-exit is set: @@ -1521,7 +1606,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { const config = niceConfig || DEFAULT_NICE_CONFIG; const cmd = wrapWithNice(baseCmd, config); const localFullCmd = `${buildNofileLimitCommand()} && ${pathExport}${envExportsStr} && ${cmd}`; - const fullCmd = remote ? buildRemoteLaunchCommand({ mode, remote, sessionId }) : localFullCmd; + const fullCmd = remote ? buildRemoteSessionCommand(mode, remote, sessionId) : localFullCmd; try { // For OpenCode: set sensitive env vars via tmux setenv before respawn @@ -1636,9 +1721,16 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { return false; } + // COD-108: an intentional kill/detach must NEVER be auto-revived by the + // remote-reconnect watcher. Guard BEFORE any teardown so a tick that fires + // mid-kill (especially the non-owned DETACH early-return below, where the + // dead local pane would otherwise look reconnectable) sees the guard. + this.guardRemoteReconnect(sessionId); + // TEST MODE: Remove from memory only — NEVER touch real tmux sessions if (IS_TEST_MODE) { this.sessions.delete(sessionId); + this.clearRemoteReconnectState(sessionId); this.emit('sessionKilled', { sessionId }); return true; } @@ -1650,6 +1742,40 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { return false; } + // COD-105 — DETACH-NOT-KILL for NON-owned remote sessions. + // + // When this session was created by ATTACHING a remote tmux session another + // Codeman owns (`remote.owned === false`), closing the tab must NOT propagate + // a remote `tmux kill-session` — that would nuke work the remote's own + // Codeman (or another instance) still relies on. We tear down ONLY the LOCAL + // pane that holds the ssh client: killing the local ssh sends SIGHUP to its + // remote `tmux attach`, which DETACHES (the durable remote session survives). + // + // This early return is the structural guarantee: no code below this point + // (now or in future for owned sessions) can ever issue a remote kill-session + // for a non-owned session. The only `kill-session` we run is on OUR LOCAL + // socket (`this.tmux()` = `tmux -L codeman` on THIS host), which kills the + // local pane — it does NOT reach the REMOTE socket. + if (session.remote && session.remote.owned === false) { + console.log(`[TmuxManager] DETACH (non-owned remote): tearing down local pane only for ${session.muxName}`); + if (isValidMuxName(session.muxName)) { + try { + // Local socket only — detaches the remote session by killing the local ssh pane. + execSync(`${this.tmux()} kill-session -t "${session.muxName}" 2>/dev/null`, { + timeout: EXEC_TIMEOUT_MS, + }); + } catch { + // Local pane may already be gone. + } + } + this.lastPaneCount.delete(session.muxName); + this.sessions.delete(sessionId); + this.clearRemoteReconnectState(sessionId); + this.saveSessions(); + this.emit('sessionKilled', { sessionId }); + return true; + } + // Get current PID (may have changed) const currentPid = this.getPanePid(session.muxName) || session.pid; @@ -1742,6 +1868,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { this.lastPaneCount.delete(session.muxName); this.sessions.delete(sessionId); + this.clearRemoteReconnectState(sessionId); this.saveSessions(); this.emit('sessionKilled', { sessionId }); @@ -1808,6 +1935,7 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { } else { dead.push(sessionId); this.sessions.delete(sessionId); + this.clearRemoteReconnectState(sessionId); this.emit('sessionDied', { sessionId }); } } @@ -2079,9 +2207,118 @@ export class TmuxManager extends EventEmitter implements TerminalMultiplexer { this.lastPaneCount.clear(); } + // ── COD-108 remote-session auto-reconnect watcher ───────────────────────── + + /** + * Start the remote-reconnect watcher (COD-108). Each tick, for every tracked + * session with `session.remote` whose local pane is DEAD, not intentionally + * guarded, and within its backoff budget, emit `remoteSessionDropped` so the + * session owner reattaches (re-running the idempotent remote command rejoins + * the durable remote tmux session). After the attempt cap, emit + * `remoteReconnectExhausted` once and go quiet. + * + * No-op tick body under `IS_TEST_MODE` (mirrors `startMouseModeSync`): tests + * drive the logic deterministically via {@link runRemoteReconnectTick}. + */ + startRemoteReconnectWatcher(intervalMs: number = DEFAULT_REMOTE_RECONNECT_INTERVAL_MS): void { + if (this.remoteReconnectInterval) { + clearInterval(this.remoteReconnectInterval); + } + this.remoteReconnectInterval = setInterval(() => { + if (IS_TEST_MODE) return; + try { + this.runRemoteReconnectTick(Date.now(), isRemoteAutoReconnectEnabled()); + } catch (err) { + console.error('[TmuxManager] Remote reconnect watcher error:', err); + } + }, intervalMs); + } + + stopRemoteReconnectWatcher(): void { + if (this.remoteReconnectInterval) { + clearInterval(this.remoteReconnectInterval); + this.remoteReconnectInterval = null; + } + } + + /** + * Run ONE watcher tick. Extracted (and given an injected `now`/`enabled`) so + * the reconnect logic is deterministically testable even though the live + * `setInterval` body no-ops under test mode. For each remote session it + * applies the pure {@link decideReconnect} decision and translates the result + * into events + backoff/state transitions. Public for tests + the watcher. + */ + runRemoteReconnectTick(now: number, enabled: boolean): void { + for (const session of this.sessions.values()) { + if (!session.remote) continue; + const sessionId = session.sessionId; + const state = this.reconnectState.get(sessionId); + const action = decideReconnect({ + session: { + sessionId, + isRemote: true, + paneDead: this.isPaneDead(session.muxName), + }, + state, + guarded: this.reconnectGuard.has(sessionId), + enabled, + now, + }); + + if (action.kind === 'emit') { + const base = state ?? freshReconnectState(); + // Mark in-flight + advance backoff BEFORE emitting so a re-entrant tick + // (or a synchronous listener) can never stack a second reconnect. + this.reconnectState.set(sessionId, { ...advanceBackoff(base, now), inFlight: true }); + this.emit('remoteSessionDropped', { sessionId, attempt: action.attempt }); + } else if (action.kind === 'exhaust') { + const base = state ?? freshReconnectState(); + if (!base.exhaustedEmitted) { + this.reconnectState.set(sessionId, { ...base, exhausted: true, exhaustedEmitted: true }); + this.emit('remoteReconnectExhausted', { sessionId }); + } + } + // 'skip' → nothing to do. + } + } + + /** + * Tell the watcher a reattach attempt for `sessionId` finished. On success, + * reset the backoff so the session is healthy again; on failure, just clear + * the in-flight flag so the next due tick can retry under the existing + * backoff schedule. Called by the session owner after `respawnPane`. + */ + noteRemoteReconnect(sessionId: string, success: boolean): void { + if (success) { + this.reconnectState.set(sessionId, resetReconnectState()); + return; + } + const state = this.reconnectState.get(sessionId); + if (state) this.reconnectState.set(sessionId, { ...state, inFlight: false }); + } + + /** + * Exclude a session from auto-reconnect (intentional teardown). Adds it to the + * guard set and drops any backoff state so a closed/killed tab — especially a + * non-owned remote DETACH — is never auto-revived. Idempotent. + */ + guardRemoteReconnect(sessionId: string): void { + this.reconnectGuard.add(sessionId); + this.reconnectState.delete(sessionId); + } + + /** Clear all per-session reconnect + guard state (e.g. when a session is removed). */ + clearRemoteReconnectState(sessionId: string): void { + this.reconnectState.delete(sessionId); + this.reconnectGuard.delete(sessionId); + } + destroy(): void { this.stopStatsCollection(); this.stopMouseModeSync(); + this.stopRemoteReconnectWatcher(); + this.reconnectState.clear(); + this.reconnectGuard.clear(); } registerSession(session: MuxSession): void { diff --git a/src/types/session.ts b/src/types/session.ts index f74d711d..3cf4d2fd 100644 --- a/src/types/session.ts +++ b/src/types/session.ts @@ -96,6 +96,45 @@ export interface SessionRemote extends RemoteSshOptions { port?: number; remotePath: string; commands?: Partial>; + /** + * COD-105 — whether THIS Codeman created the remote tmux session. + * + * - `true` (default for COD-104 launched sessions): we own the remote session; + * an explicit "kill" may propagate a remote `tmux kill-session`. + * - `false` (discovered + attached an existing remote session another Codeman + * created): closing the local tab must DETACH only — we must NEVER issue a + * remote `kill-session`, or we'd nuke work the remote's own Codeman (or + * another instance) still relies on. See `killSession()` gate. + * + * Absent is treated as owned (legacy/COD-104 sessions persisted before this + * field existed were all launched by us). + */ + owned?: boolean; + /** + * COD-105 — for a NON-owned (discovered + attached) session, the EXISTING + * remote tmux session name to `attach -t` (e.g. `codeman-disco1`). It differs + * from this Codeman's deterministic `codeman-` name because the remote + * session was created elsewhere. Only meaningful when `owned === false`. + */ + remoteSessionName?: string; +} + +/** + * COD-105 — a `codeman-*` tmux session discovered on a remote host's + * `tmux -L codeman` socket (may have been created by the remote's own Codeman, + * another instance, or this one). Returned by `listRemoteCodemanSessions`. + */ +export interface RemoteSessionInfo { + /** tmux session name (always starts `codeman-`). */ + name: string; + /** Whether at least one client is currently attached to the remote session. */ + attached: boolean; + /** COD-106 — number of clients attached (tmux `session_attached`); >1 = shared. */ + attachedClients: number; + /** tmux `session_created` epoch seconds. */ + created: number; + /** Number of windows in the remote session. */ + windows: number; } /** diff --git a/src/web/public/app.js b/src/web/public/app.js index c27c7f48..fb7c292c 100644 --- a/src/web/public/app.js +++ b/src/web/public/app.js @@ -216,6 +216,10 @@ const _SSE_HANDLER_MAP = [ [SSE_EVENTS.MUX_DIED, '_onMuxDied'], [SSE_EVENTS.MUX_STATS_UPDATED, '_onMuxStatsUpdated'], + // Remote auto-reconnect (COD-108) + [SSE_EVENTS.REMOTE_SESSION_RECONNECTED, '_onRemoteSessionReconnected'], + [SSE_EVENTS.REMOTE_RECONNECT_EXHAUSTED, '_onRemoteReconnectExhausted'], + // Ralph [SSE_EVENTS.SESSION_RALPH_LOOP_UPDATE, '_onRalphLoopUpdate'], [SSE_EVENTS.SESSION_RALPH_TODO_UPDATE, '_onRalphTodoUpdate'], diff --git a/src/web/public/constants.js b/src/web/public/constants.js index 1751bfe3..8d57b026 100644 --- a/src/web/public/constants.js +++ b/src/web/public/constants.js @@ -379,6 +379,11 @@ const SSE_EVENTS = { MUX_DIED: 'mux:died', MUX_STATS_UPDATED: 'mux:statsUpdated', + // Remote auto-reconnect (COD-108) + REMOTE_SESSION_DROPPED: 'remote:sessionDropped', + REMOTE_SESSION_RECONNECTED: 'remote:sessionReconnected', + REMOTE_RECONNECT_EXHAUSTED: 'remote:reconnectExhausted', + // Ralph SESSION_RALPH_LOOP_UPDATE: 'session:ralphLoopUpdate', SESSION_RALPH_TODO_UPDATE: 'session:ralphTodoUpdate', diff --git a/src/web/public/index.html b/src/web/public/index.html index 362e9a07..62d6ed9c 100644 --- a/src/web/public/index.html +++ b/src/web/public/index.html @@ -1471,6 +1471,14 @@

App Settings

Use 1M token context window (model: opus[1m]) for all new sessions — ignored when a Claude Model is selected above +
+ + + Automatically re-establish remote (SSH) sessions when the connection drops, reattaching to the durable remote tmux session (on by default; bounded backoff) +