Skip to content

Add a memory-pressure guard to executor spawns (warn by default)#681

Merged
bborn merged 3 commits into
mainfrom
feat/memory-pressure-spawn-guard
Jul 25, 2026
Merged

Add a memory-pressure guard to executor spawns (warn by default)#681
bborn merged 3 commits into
mainfrom
feat/memory-pressure-spawn-guard

Conversation

@bborn

@bborn bborn commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Why

Nothing in ty bounded how many agent sessions could be live at once. Each session is a claude process plus its MCP servers, so a queue that fans out freely can walk a workstation into swap thrashing — at which point every session, dev server and test run on the box gets slower, including the ones already doing useful work.

Measured on a 24GB M4 while diagnosing this: 36 live sessions across 393 processes, of which 35 session trees belonged to tasks that were already done or blocked — 4.4GB resident and a large share of what was filling 27GB of swap. Only 2 tasks were actually processing.

What this is not

Deliberately not a concurrency cap. A fixed "max N sessions" is a hidden wall — the right number depends entirely on how much RAM the machine has and what else is running, and someone who legitimately wants dozens of concurrent agents shouldn't be told "no" by an arbitrary constant.

So this gates on the machine's actual memory pressure instead. With headroom, spawn as many as you like. The guard only has an opinion once the kernel says memory is genuinely short.

How

systemFreeMemoryPct is implemented per platform behind one stable contract — percent of memory still available, or "can't tell" — so the threshold means the same thing everywhere:

Platform Signal
darwin kern.memorystatus_level — what Activity Monitor's memory-pressure graph is derived from
linux cgroup v2 if the cgroup is limited, else /proc/meminfo MemAvailable
everything else inert — guard has no opinion, spawns proceed as before
TY_MEMORY_GUARD=off|warn|block        # default: warn
TY_MEMORY_GUARD_MIN_FREE_PCT=<0-100>  # default: 20

On macOS the default of 20 maps to Activity Monitor's zones (green 100-50, yellow 50-30, red 30-0), so it fires only well into the red — when the machine is already hurting, not merely busy.

Default mode is warn: it logs loudly and never blocks. Behaviour is unchanged for everyone unless they opt in. TestGuardMemoryForSpawnDefaultNeverBlocks pins that property specifically — it forces the threshold to 100% (so the machine is guaranteed "under pressure") and asserts the default mode still returns no error.

In block mode the spawn returns ErrMemoryPressure carrying the actual free percentage, the threshold, the task id, and the override — so if it ever does bite, it says why and how to get past it rather than silently stalling.

Wired into both spawn choke points: createTmuxWindow (daemon path) and EnsureTaskWindow (TUI/API path).

Two Linux details that matter

Both are load-bearing, and both have tests:

  • MemAvailable, not MemFree. Linux fills RAM with reclaimable cache, so MemFree reads ~1% on a perfectly healthy box. Using it would wedge spawns in block mode for no reason. Pre-3.14 kernels fall back to MemFree + Buffers + Cached.
  • cgroup v2 first, and subtract inactive_file. A containerised agent server must be measured against its own limit, not the host's RAM — otherwise a 2GB container on a 64GB host looks infinitely roomy right up until the OOM killer fires. And because memory.current counts page cache, which is reclaimable, a long-running container would otherwise sit at 0% free forever and stop spawning entirely.

PSI (/proc/pressure/memory) is deliberately not used. It's the sharper Linux pressure signal, but it measures stall time rather than a fraction of memory, so it can't share TY_MEMORY_GUARD_MIN_FREE_PCT's meaning with the macOS path. One threshold that means the same thing on every platform seemed worth more than a slightly better Linux signal.

Testing

gofmt and go vet clean; full internal/executor suite passes. Verified building for darwin, linux, freebsd and openbsd. (Windows still doesn't build, on executorlock's syscall.Flock — pre-existing, confirmed by building the branch without these files, and untouched here.)

17 tests. The Linux parsers live in a build-tag-free file specifically so they're unit-testable on the macOS machines this gets developed on, rather than shipped unverified:

  • Guard logic: mode parsing (unset and unrecognised both fall back to warn, never block), threshold parsing and out-of-range fallback, off being inert, the default-never-blocks property, block mode at 100% and 0% thresholds, raw signal in range.
  • meminfo: normal parse, the MemAvailable-vs-MemFree property, pre-3.14 fallback, SwapCached not leaking into Cached, garbage rejection.
  • cgroup: normal parse, page-cache-is-available property, unlimited (max) falling back, garbage rejection, over-limit clamping.

Tests skip rather than fail where the live signal is unavailable, so they're safe on any CI.

Follow-up not included here

A --strict-mcp-config change was considered to cut per-session MCP fan-out (~21 servers merge into every task session today, only 1 of which is taskyou). It was rejected: strict mode's mechanism is "ignore everything except this file," which breaks ty's premise that a worktree agent has the same MCP servers its parent repo has. Worth revisiting only if Claude Code grows a way to disable user/plugin scope while keeping project scope.

🤖 Generated with Claude Code

bborn and others added 3 commits July 25, 2026 10:53
Nothing bounded how many agent sessions could be live at once. Each session
is a claude process plus its MCP servers, so a queue that fans out freely can
walk a workstation into swap thrashing — at which point every session, dev
server and test run on the box gets slower, including the ones already doing
useful work. Observed on a 24GB machine: 36 live sessions, 393 processes, of
which 35 belonged to tasks already done or blocked.

Deliberately NOT a concurrency cap. A fixed "max N sessions" is a hidden wall:
the right number depends entirely on the machine and what else is running, and
someone who legitimately wants dozens of concurrent agents shouldn't be told no
by an arbitrary constant. This gates on actual memory pressure instead — with
headroom, spawn as many as you like; the guard only has an opinion once the
kernel says memory is genuinely short.

The signal is Darwin's kern.memorystatus_level, the same free-memory
percentage Activity Monitor's pressure graph derives from. Its zones are green
100-50, yellow 50-30, red 30-0, so the default threshold of 20 fires only well
into the red. Where the signal is unavailable the guard is inert.

Default mode is "warn": log loudly, never block. Behaviour is unchanged for
everyone unless they opt in, and a test pins that property specifically.

  TY_MEMORY_GUARD=off|warn|block        (default: warn)
  TY_MEMORY_GUARD_MIN_FREE_PCT=<0-100>  (default: 20)

In block mode the spawn returns ErrMemoryPressure carrying the actual numbers
and the override, so if it ever does bite it says why and how to get past it
rather than silently stalling.

Wired into both spawn choke points: createTmuxWindow (daemon) and
EnsureTaskWindow (TUI/API).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard was Darwin-only: on Linux the kern.memorystatus_level sysctl fails,
systemFreeMemoryPct reports "unknown" and the guard silently does nothing. That
is the safe failure mode, but it left the agent-server fleet — which is where
an unattended fan-out is least likely to be noticed — with no protection at all.

systemFreeMemoryPct is now per-platform behind a stable contract ("percent of
memory still available, or ok=false"), so the threshold means the same thing
everywhere:

  darwin      kern.memorystatus_level (what Activity Monitor's graph shows)
  linux       cgroup v2 if limited, else /proc/meminfo MemAvailable
  everything  else inert, exactly as before

Two Linux details that matter, both covered by tests:

  - MemAvailable, not MemFree. Linux fills RAM with reclaimable cache, so
    MemFree reads ~1% on a perfectly healthy box. Using it would have wedged
    spawns in block mode for no reason.
  - cgroup v2 is consulted first so a containerised agent server is measured
    against its own limit, not the host's RAM — otherwise a 2GB container on a
    64GB host looks infinitely roomy right up until the OOM killer fires. And
    inactive_file is subtracted from memory.current, because page cache is
    reclaimable; without that a long-running container sits at 0% free forever.

PSI (/proc/pressure/memory) is deliberately not used: it is the sharper Linux
pressure signal, but it measures stall time rather than a fraction of memory, so
it cannot share TY_MEMORY_GUARD_MIN_FREE_PCT's meaning with the macOS path. One
threshold that means the same thing on every platform is worth more here.

The parsers are in a build-tag-free file so they are unit-testable on macOS
rather than shipped unverified; 11 new tests cover them. Verified building for
darwin, linux, freebsd and openbsd. (Windows still fails to build, on
executorlock's syscall.Flock — pre-existing and untouched by this change.)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
govulncheck flags an infinite loop on invalid input in x/text v0.37.0
(GO-2026-5970), fixed in v0.39.0. It is an indirect dependency, but govulncheck
finds a reachable symbol trace, so it reports as affecting this module.

Pre-existing on main — the Vulnerability Check job started failing on 2026-07-22
and passed on 07-21, so the advisory landed in between. Folded in here rather
than left for a separate PR so CI on this branch is clean.

Only x/text moves in go.mod; the additional go.sum entries are module-graph
checksums pulled in by x/text's own go.mod, not new build dependencies.

Verified with the same govulncheck the workflow pins (v1.3.0): the x/text
finding is gone and no non-stdlib findings remain. The stdlib findings that
still show locally are an artifact of a local go1.25.5 toolchain — CI sets
check-latest so it builds against the newest patched 1.25.x.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@bborn
bborn merged commit dfc99e5 into main Jul 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant