feat: add cross-worktree device ownership and safe recovery

[thymikee]

Summary

Add a platform-neutral device ownership layer so agents can discover, acquire, release, and safely recover devices across worktrees without inspecting or killing PIDs.

The intended experience is:

open -> use -> close

busy?
  -> device status
  -> reuse the owning session, choose another device,
     release --stale, or explicitly release --force

The same public behavior must apply to local and remote iOS and Android targets. Platform-specific resources such as XCTest runners, Android instrumentation, the test IME, recordings, provider allocations, and daemon processes remain implementation details behind that interface.

Motivation

Ownership currently exists at several different layers:

• SessionStore prevents two sessions in one daemon from using the same device, but each source checkout normally has its own state directory and daemon.
• Remote device leases support iOS and Android, with TTL and heartbeat semantics, but have no public list/release CLI surface.
• Apple runner leases prevent cross-daemon XCTest ownership, but contention appears late as runner-preparation COMMAND_FAILED rather than structured DEVICE_IN_USE at open.
• Local Android has no equivalent host-global device claim. Two worktree daemons can reach the same device and conflict later through instrumentation/UiAutomation, IME restoration, recordings, or ADB resources.
• session list only reports sessions visible to the selected daemon/workspace scope, so it can disagree with ownership errors from another daemon.
• Recovery guidance can degrade into ps, kill, sleeps, and runner PID checks even though much of the required identity and cleanup machinery already exists.

Related history:

• Physical iOS commands can time out in activate/wait-for-idle before executing, leaving hidden stale device ownership #1246 records the user-visible DEVICE_IN_USE / hidden-owner failure shape.
• fix: reap idle daemons and take over stale runner leases #1169 added idle daemon reap and proof-based stale Apple runner takeover.

Goals

• Enforce exclusive device ownership across local worktree daemons.
• Use the same public ownership lifecycle for iOS and Android.
• Make ownership inspectable without starting a daemon.
• Automatically reclaim only provably orphaned ownership.
• Provide graceful-first explicit recovery for live owners.
• Keep device claims, remote leases, daemon sessions, and runner/process leases as distinct concepts.
• Prevent device claims from becoming free while attributable resources remain owned.
• Preserve warm Apple runner reuse where it does not conflict with logical ownership.
• Give agents structured recovery data instead of requiring prose parsing or PID discovery.

Non-goals

• Merge remote leases and Apple runner leases into one record.
• Permit concurrent mutating sessions on one device.
• Reclaim a verified live owner because it appears inactive.
• Treat deleting a claim file as resource cleanup.
• Add per-command host-global filesystem reads or writes to session-bound hot paths.
• Expose daemon tokens or provider credentials in claim records.

Public commands

Use case	Command	Behavior
Inspect all ownership	agent-device device status	Reads all local claims without starting a daemon; includes remote lease status when a connection is configured.
Inspect one device	agent-device device status <device selectors>	Shows owner session/worktree/state dir, liveness classification, resources, and exact recovery command.
Release own/reachable session	agent-device device release <selectors or --session>	Routes through ordinary session close and releases the claim last.
Reclaim a dead owner	agent-device device release <selectors> --stale	Succeeds only with positive proof of orphaned ownership and runs platform reconciliation.
Interrupt a live owner	agent-device device release <selectors> --force	Contacts the owner daemon first; falls back to verified owner-scoped cleanup.
Stop a daemon	agent-device daemon stop --state-dir <path>	Verified graceful stop with bounded escalation; detached simulator runners may remain adoptable.
Stop and scrub a daemon	agent-device daemon stop --state-dir <path> --clean	Also removes retained/detached runners and their leases.

Existing flat devices, boot, shutdown, and capabilities commands remain compatible. devices is plural inventory; the singular device group owns status/release lifecycle.

Terminology and authority

Device claim

Host-global exclusive ownership of a local provider-scoped device by an agent-device session or transient mutating command.

Local claims live under a user-private host-global directory such as:

~/.agent-device/device-claims/

Claim keys must be derived from canonical platform/device identity rather than a hardcoded iOS/Android list, covering registered local targets such as iOS, Android, tvOS, and macOS.

Remote device lease

Existing daemon/provider-authoritative logical ownership with TTL/heartbeat semantics. Remote targets do not create local claim files. Their provider/device keys are response projections only.

Runner/process lease

Existing Apple implementation-level mutual exclusion for XCTest. It may outlive the device claim when a runner is retained for reuse.

Claim record and shared primitives

The device claim should generalize proven owner mechanics from runner-lease.ts, extracting shared helpers rather than creating a second staleness implementation:

• PID plus process start-time identity;
• owner state directory;
• atomic tmp+rename writes;
• process-lock serialization;
• owner-process-dead versus owner-state-dir-gone classification;
• statSync proof of absence using only ENOENT/ENOTDIR;
• fail-closed behavior for permission and transient filesystem errors.

Illustrative session claim:

{
  "schemaVersion": 1,
  "kind": "session",
  "deviceKey": "local:android:emulator-5554",
  "platform": "android",
  "session": "qa",
  "workspace": "/worktrees/feature-a",
  "stateDir": "/Users/me/.agent-device/dev/feature-a-a81b2c",
  "ownerPid": 87193,
  "ownerStartTime": "...",
  "claimedAt": 1784276460000
}

Transient claims include kind: "transient" and the command name. Claims are updated only on lifecycle transitions, never on every command.

Ownership states

State	Meaning	Acquisition/recovery
free	No owner	Acquire normally.
claimed	Verified live session or transient owner	Refuse with DEVICE_IN_USE, retriable: false.
releasing + live owner	Teardown in progress	Refuse or retry after a bounded delay.
releasing + dead owner	Teardown interrupted	Classify orphaned and reconcile.
orphaned	PID dead/reused or state dir provably gone	Eligible for automatic or --stale reconciliation.
inconsistent	Corrupt or contradictory record	Fail closed unless independent positive evidence proves staleness; otherwise require --force.
release-failed	Cleanup did not complete	Preserve ownership and report failed steps.

Elapsed inactivity alone never makes a live owner reclaimable.

Corrupt claim rules

• Inspection has no side effects: open and device status never rename, quarantine, or free corrupt claims.
• Absence from a scan of known daemons is not proof of staleness because daemons may use arbitrary state directories.
• Scan results may confirm a live owner but cannot prove no owner exists.
• Quarantine occurs only inside an authorized recovery transaction, under the claim lock.
• --stale may quarantine only when partially recoverable identity provides positive stale proof.
• Otherwise explicit --force is required.
• Missing process-lock owner metadata may use the existing short grace-window cleanup; that rule does not apply to finalized claim records.

Command descriptor policy

Every command descriptor must declare a required policy with no default:

type DeviceClaimPolicy =
  | 'none'
  | 'observe'
  | 'require-owner'
  | 'transient-exclusive'
  | 'acquire-session'
  | 'release-session';

Policy	Example commands	Enforcement
none	Host/config-only commands	No claim interaction.
observe	devices, capabilities, device status	May project ownership, never mutate it.
require-owner	Session-bound snapshots/interactions/recording	Trust the daemon invariant established by open; no per-command claim-store I/O.
transient-exclusive	boot, shutdown, sessionless device mutations	Acquire a command-scoped claim and release in finally; refuse foreign claims.
acquire-session	open	Acquire before platform preparation or mutation.
release-session	close	Release only after teardown reaches a safe terminal state.

Registry guards must fail when a new command lacks an explicit policy. Audit all sessionless device-mutating commands; do not stop at boot/shutdown.

Lifecycle

Local open

resolve canonical device
-> inspect/reconcile claim under lock
-> automatically reclaim only when provably orphaned
-> acquire session claim
-> reconcile subordinate platform resources
-> prepare runner/helper
-> perform open
-> commit session

Failure after claim acquisition cleans partial resources and releases the claim.

Remote open

Use the remote daemon/provider lease authority. No host-local claim file is written on the client.

Close/release

lock claim
-> verify release authority
-> mark releasing
-> stop/finalize session resources
-> release provider allocation when applicable
-> delete session
-> clear claim last

If cleanup fails, preserve release-failed or claimed state with structured failed steps.

Apple runner interaction

Resolve retained-runner behavior in v1:

Holding a valid device claim authorizes reclaiming an Apple runner whose owner no longer holds that device claim.

Use stop-and-recreate in v1, not live adoption:

1. Daemon A closes and releases the device claim but may retain a warm runner.
2. Daemon B acquires the now-free claim.
3. B finds A's live runner lease.
4. Since A no longer owns the claim, B stops A's runner and starts its own.
5. A's later cleanup re-reads the runner lease and no-ops after token mismatch.

A live runner owner that still owns the device claim remains protected. This moves iOS contention to structured DEVICE_IN_USE during open instead of late runner-prep COMMAND_FAILED.

Android reconciliation

For a proven-dead local Android owner:

1. Reconfirm stale identity under the claim lock.
2. Force-stop the actual recorded helper package when known.
3. Fall back to the bundled helper package com.callstack.agentdevice.snapshothelper for local/default helpers.
4. Restore the device-persisted previous IME only when the agent-device test IME remains active.
5. Stop attributable recording/profiling processes.
6. Reconcile known ADB resources.
7. Clear the claim only after reconciliation succeeds.

Provider-supplied helpers use provider-owned cleanup. Local session-owned ADB reverse cleanup is a separate prerequisite/gap and must not be described as existing behavior.

Owner-daemon transport

Local device status reads claim files directly and works with no daemon running.

Graceful release of a foreign live owner uses:

claim -> owner stateDir -> protected daemon.json -> ordinary close request

No transport token is duplicated into the claim. --force first attempts this graceful close, then performs only verified owner-scoped termination if the owner cannot cooperate.

Error contract

Live owner conflicts must override the code-level retriable default:

{
  "code": "DEVICE_IN_USE",
  "message": "Android device emulator-5554 is owned by session \"qa\" in another worktree.",
  "retriable": false,
  "details": {
    "reason": "DEVICE_CLAIM_LIVE_OWNER",
    "classification": "live-owner",
    "deviceKey": "local:android:emulator-5554",
    "owner": {
      "session": "qa",
      "workspace": "/worktrees/feature-a",
      "stateDir": "/Users/me/.agent-device/dev/feature-a-a81b2c",
      "daemonPid": 87193,
      "daemonStartTime": "..."
    },
    "recovery": {
      "command": "agent-device device release --platform android --serial emulator-5554 --force"
    }
  }
}

Expiry-based remote contention may remain retriable and should expose expiresAt.

Stage 0: daemon stop and shutdown lease finalization

Stage 0 is independently useful and requires no claim infrastructure.

Current explicit close is the only production caller of releaseSessionLease. Daemon shutdown runs resource teardown, finalizes repair state, and deletes sessions without routing each active lease through that release lifecycle. Provider runtimes attempt to terminate active sessions during shutdown, including happy-path Limrun instance deletion, but a failed active provider deletion is not durably captured for retry before runtime/session state is cleared.

Implement:

• daemon stop --state-dir <path>: verify daemon PID/start time, send SIGTERM, wait for graceful shutdown/metadata removal, escalate to SIGKILL only after a bounded timeout.
• daemon stop --clean: additionally remove retained/detached runner processes and leases.
• During graceful shutdown, explicitly finalize every active session lease/provider allocation before deleting session state.
• Perform a bounded final drain of pending and newly failed provider releases.
• Persist unresolved provider identity only on the graceful path, while the daemon is alive to write it.
• If SIGKILL is required, report provider state as unknowable; do not claim enumeration.

Proxy leases are asymmetric: their remote daemon remains authoritative and reaps them after heartbeats stop and TTL expires. Locally hosted provider runtimes attempt active cleanup, but failed deletion durability is the gap Stage 0 closes.

Illustrative forced result:

{
  "stopped": true,
  "mode": "forced",
  "cleanupConfidence": "unknown",
  "claimsReleased": [],
  "claimsOrphaned": [],
  "providerReleases": {
    "status": "unknown",
    "released": [],
    "pending": null
  },
  "warnings": [
    "The daemon was force-killed before provider lease state could be finalized. Provider allocations may remain active."
  ]
}

pending: null means unknowable; pending: [] means known and empty. claimsReleased and claimsOrphaned ship empty in Stage 0 for response-shape stability.

After claims exist:

• claimsReleased means stale ownership was proven and platform cleanup completed.
• claimsOrphaned means staleness was proven but cleanup remains pending.
• PID death alone moves a claim to orphaned, not released.

Allocation-time provider journaling is deferred. If forced-stop evidence later justifies that additional crash-consistency surface, track it separately.

Rollout

Stage	Deliverables	Enforcement
0	daemon stop [--clean], shutdown-time active lease finalization, bounded provider drain, confidence-graded result	None
1	Shared owner classification, advisory local claim writes, daemonless device status	Disabled
2	Required descriptor policies, atomic enforcement, --stale, platform reconciliation, automatic proven-orphan reclaim	Feature-gated
3	Owner-daemon transport and device release --force	Feature-gated
4	Remote lease projection with provider/backend/expiresAt	Enable after soak
5	Remove feature gate and retire manual PID-kill guidance	Required

Keep the feature gate through Stage 2 so enforcement can be disabled without reverting advisory claim writes and inspection.

Acceptance criteria

• Two worktrees cannot open the same local Android or Apple device simultaneously.
• Contention is rejected before platform mutation.
• A third worktree can inspect ownership without starting a daemon.
• A live-owner conflict is structured DEVICE_IN_USE with retriable: false and an exact next command.
• session list and ownership errors no longer leave an undiscoverable owner.
• Dead owners are automatically reclaimable only with positive proof.
• --stale never interrupts a verified live owner.
• --force attempts owner-daemon graceful close first.
• Sessionless device mutations cannot affect a foreign claimed device.
• No per-command host-global claim read/write is added to session-bound commands.
• A free device with a retained Apple runner can be acquired; the claim-holding daemon stops/recreates the old runner.
• Android stale recovery stops helper instrumentation and restores the test IME before clearing the claim.
• Corrupt finalized claims remain fail-closed and inspection-only operations have no side effects.
• releasing + dead owner is stale-recoverable; releasing + live owner remains protected.
• Cleanup failures preserve recoverable ownership state.
• Graceful daemon stop enumerates released/pending provider allocations.
• Forced daemon stop reports providerReleases.pending: null and never implies known cleanup.
• Claim results distinguish reconciled claimsReleased from cleanup-pending claimsOrphaned.

Sequencing and follow-ups

• Stage 0 is the first self-contained implementation and immediately replaces manual PID-kill guidance.
• Local Android session-owned ADB reverse cleanup is an independent prerequisite for complete Android stale reconciliation.
• This issue is an umbrella design/rollout issue; implementation may be split into one child issue per stage, preserving the ordering above.

[thymikee]

Stage 0 complete ✅

Stage 0 shipped in #1323 (630e0bb65, merged 2026-07-17).

Delivered:

• agent-device daemon stop --state-dir <path> with verified graceful shutdown and bounded forced escalation
• --clean cleanup for retained daemon-owned resources
• shutdown-time finalization of active session/provider leases before session deletion
• bounded draining and durable reporting of unresolved provider releases
• confidence-graded graceful vs forced-stop results, including unknown provider state after forced termination

The umbrella can now proceed to Stage 1: shared owner classification, advisory local claim writes, and daemonless device status.

[thymikee]

Stage 1 is now split into #1328: advisory host-global device claims, shared owner classification, and daemonless device status. It is labeled ready-for-agent; enforcement and recovery remain Stage 2+.