Skip to content

feat: tmux-backed session persistence — terminals survive server restarts#1027

Open
web3dev1337 wants to merge 5 commits into
mainfrom
feature/tmux-session-persistence
Open

feat: tmux-backed session persistence — terminals survive server restarts#1027
web3dev1337 wants to merge 5 commits into
mainfrom
feature/tmux-session-persistence

Conversation

@web3dev1337

Copy link
Copy Markdown
Owner

Summary

Terminals now run inside per-session tmux sessions on a dedicated per-instance socket (agent-workspace-<port>); the pty the orchestrator owns is only a tmux client. When the app server restarts — nodemon reload, version update, or crash — the panes keep running under the tmux server, and the next createSession() for the same deterministic id re-attaches via new-session -A. Live agents survive.

Closes the "hot reload without losing sessions" half of #1025. (Reboots still rely on transcript resume — that's the other half.)

Why / how

  • Env hygiene at the tmux-server choke point: CLAUDECODE, CLAUDE_CODE_ENTRYPOINT, and TMUX are scrubbed so the CLI's nested-session guard never trips, regardless of how the orchestrator itself was launched (the exact trap we hit with ORCHESTRATOR_PORT this cycle).
  • Panes behave like plain terminals: status bar off, prefix key off (Ctrl-B reaches the agent), tmux mouse off (a pane's own mouse-mode requests still pass through), history-limit 20000, window-size latest, 256-color outer TERM.
  • Kill vs detach semantics: explicit close / terminate / workspace teardown kills the tmux session so nothing leaks detached; server shutdown only detaches — that's the survival feature. Process-tree kills and the per-session process limit now target the pane pid (a child of the tmux server), which the old client-pid group-kill could never reach.
  • Adoption backfills the session buffer from capture-pane (escape codes preserved) so the log endpoint / scrollback preload shows pre-restart history, marked delivered to avoid double-streaming.
  • Fail-closed fallback: Windows and tmux-less installs keep the direct node-pty path unchanged (packaged Tauri builds unaffected). Disable via ORCHESTRATOR_SESSION_PERSISTENCE=0 or config.sessions.persistence.enabled=false.
  • Observability: GET /api/sessions/persistence reports managed vs orphaned sessions; orphans stay inspectable via tmux -L agent-workspace-<port> attach -t <name>.

Test plan

Unit: 16 new tests (backend argv/quoting/env-scrub/config/target-matching; session-manager pane-pid resolution, kill-vs-detach, orphan reporting). Full suite 617/617 green.

End-to-end on a live instance (34 real sessions, port 3000):

  1. Wrote a marker into a session through the orchestrator; recorded pane pids.
  2. Killed the server process, then triggered a real nodemon reload (touch server file).
  3. After restart: pane pids identical (1398113 / 1398185 unchanged) → panes genuinely survived.
  4. Pre-restart marker still present in scrollback → adoption + backfill work.
  5. Persistence status: 34 managed, 0 orphaned → clean re-adoption of every session.
  6. Sent a fresh command post-restart → executed → full interactivity preserved.

Tauri / Windows note

tmux is Linux/WSL/macOS only, so the packaged Windows build keeps the in-process model (feature auto-disables). A follow-up can add a broker-daemon equivalent for native Windows if desired; tracked in #1025.

🤖 Generated with Claude Code

web3dev1337 and others added 2 commits July 19, 2026 18:23
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…arts

Terminals now run inside per-session tmux sessions on a dedicated
per-instance socket; the pty the orchestrator owns is only a tmux client.
When the app server restarts (nodemon reload, update, crash) panes keep
running under the tmux server, and the next createSession() for the same
deterministic id re-attaches via `new-session -A` — live agents survive.

Design points (issue #1025):
- Env hygiene at the tmux-server choke point: CLAUDECODE /
  CLAUDE_CODE_ENTRYPOINT / TMUX are scrubbed so nested-session guards
  never trip regardless of how the orchestrator itself was launched.
- Socket options make panes act like plain terminals: status off, prefix
  None, tmux mouse off (pane mouse-mode requests still pass through),
  history-limit 20000, window-size latest, 256-color outer TERM.
- Kill vs detach semantics: explicit close/terminate/workspace teardown
  kills the tmux session so nothing leaks detached; server shutdown only
  detaches — that is the survival feature. Tree-kills and the per-session
  process limit now target the PANE pid (a child of the tmux server),
  which the old client-pid group kill could never reach.
- Adoption backfills the session buffer from capture-pane (escape codes
  preserved) so the log endpoint / scrollback preload shows pre-restart
  history, marked delivered to avoid double-streaming.
- Fail-closed fallback: Windows and tmux-less installs keep the direct
  node-pty path unchanged (packaged Tauri builds unaffected); disable via
  ORCHESTRATOR_SESSION_PERSISTENCE=0 or config.sessions.persistence.
- GET /api/sessions/persistence reports managed vs orphaned sessions;
  orphans stay inspectable via `tmux -L agent-workspace-<port> attach`.

Tests: 16 new unit tests (backend argv/quoting/env-scrub/config, session
manager pid resolution, kill semantics, orphan reporting); full suite
617/617 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
web3dev1337 and others added 2 commits July 20, 2026 10:23
…;276;0c)

Under tmux, the browser terminal's auto-response to tmux's outer-terminal
capability probe (DA1 ESC[?1;2c / DA2 ESC[>0;276;0c, 276 = xterm.js version)
round-trips back over the socket and can arrive on the tmux client's stdin
after tmux's read window closes; tmux then routes the stray bytes to the
active pane, where they surface as "1;2c0;276;0c" at the shell prompt.
Only Claude/TUI sessions were affected (they trigger the capability
negotiation); non-tmux instances never showed it.

Two-part fix:
- Pin the outer terminal's capabilities via `terminal-features`
  (xterm-256color:RGB:clipboard) so tmux never depends on the (slow,
  browser-round-tripped) probe response to detect truecolor/clipboard.
- Strip DA report sequences (ESC[?...c / ESC[>...c) from inbound input when
  a session is tmux-backed. These are terminal auto-responses a user never
  types; tmux answers inner-app DA queries itself, so nothing legitimate
  depends on the echo. Cursor-position (...R) and DSR (...n) reports are
  deliberately preserved — some apps read those back as input.

Verified: tmux self-answers inner DA queries (repro), so filtering the
echo is safe. Adds 3 unit tests for stripDeviceReports; 19 persistence
tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oint

commanderService.sendToSession wrote to session.pty directly, bypassing
writeToSession — so the device-report stripping (and PowerShell CRLF
normalization, activity/status bookkeeping) didn't apply to that path.
Route it through writeToSession so every input source shares one choke
point; keep the direct write only as a fallback. Adds writeToSession
device-report unit tests (tmux strips, direct does not).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@web3dev1337

Copy link
Copy Markdown
Owner Author

Follow-up fixes after live testing surfaced a tmux-only cosmetic regression: device-attribute reports leaking as 1;2c0;276;0c junk at the shell prompt.

Root cause (verified, not guessed): tmux answers inner-app DA queries itself, but it also probes the outer terminal (xterm.js, whose DA2 version is 276) for capabilities. That probe response round-trips over the browser socket and can arrive on the tmux client's stdin after tmux's read window closes; tmux then routes the stray bytes to the active pane. Only Claude/TUI sessions triggered it (they drive the capability negotiation); non-tmux instances never showed it.

Fix (three parts, all shipped on this branch):

  1. Pin outer-terminal capabilities via terminal-features xterm-256color:RGB:clipboard so tmux never depends on the (slow, round-tripped) probe response.
  2. Strip DA report sequences (ESC[?…c / ESC[>…c) from inbound input when a session is tmux-backed — these are terminal auto-responses a user never types, and tmux self-answers inner-app DA, so nothing legitimate depends on the echo. Cursor-position (…R) and DSR (…n) reports are deliberately preserved.
  3. Routed commanderService.sendToSession through writeToSession so every input path shares one choke point (it was writing to the pty directly and bypassing the filter).

Verified end-to-end on the live 34-session test instance: sending ESC[?1;2cESC[>0;276;0c<marker> through the real input path now yields only <marker> with zero 1;2c/0;276;0c junk (same test pre-fix produced the junk). Full suite 622/622 green. Note: pre-existing junk already in a session's scrollback stays until it scrolls off; the fix stops new occurrences.

Two app-side console sources that get loud with many worktrees, both
describing normal operation rather than problems:
- Per-worktree button-config lookups logged "using defaults" for every
  worktree without a custom .orchestrator-config.json (the common case).
  Gated behind localStorage 'debug-worktree-config'.
- The fit retry warned "proposed fit still too small (0x0)" for terminals
  that are simply off-screen in the scrollable grid (unmeasurable, not a
  real failure). The exact 0x0 case is now debug-level and schedules a
  delayed retry so it fits when scrolled into view; genuine small-but-
  nonzero fit failures still warn.

(The unrelated web-client-content-script.js errors are from a browser
extension, not this app.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant