Client cluster doesn't survive a laptop restart — local-path PVCs strand Pending after WSL2/Docker restart

[shujaatTracebloc]

Follow-up from the jobs-manager RWO deadlock incident (#545).

Observation

The incident was triggered by a routine WSL2/Docker/laptop restart ~4h before: all kube-system pods restarted at once, and afterwards the three local-path PVCs (client-pvc, client-logs-pvc, mysql-pvc) were stuck Pending (empty Capacity) and never re-bound. A sleep/reboot should never strand a client.

Investigate / harden

• Does local-path storage persist across a k3d cluster stop/start and a Docker Desktop/WSL2 restart? Confirm the installer mounts a persistent host volume for the node's storage path so PVs survive.
• Does a restarted cluster self-heal to Bound with a single stable consumer (post jobs-manager RollingUpdate on RWO PVCs deadlocks single-node local-path after a node restart #545 Recreate)?
• Add a node-restart → re-provision cycle to the auto-upgrade / cluster e2e so this is covered.
• Consider a restart unless-stopped / autostart check on Windows/WSL parity with the Linux/macOS reboot story.

[shujaatTracebloc]

Investigation (from the #545 incident) — this is a storage-DESIGN decision, not a bug

The client's PVCs were on the local-path StorageClass = node-local mode. Our own cluster-creation code documents node-local as ephemeral:

scripts/lib/cluster.sh (_create_new_cluster): "node-local model (RFC-0003 Option C): no host bind-mount, and KEEP k3s local-storage so its local-path StorageClass provisions the dataset volumes inside the node — data then dies with the cluster."

So the data loss on restart is by design of node-local mode, not an accidental bug. The default is hostpath (bind-mounts ~/.tracebloc:/tracebloc, persists on the host); this client was on node-local.

This is entangled with #456 (RFC-0003 D15: flip the default TO node-local). If node-local becomes the default, its non-durability across a node recreation becomes unacceptable and must be solved first.

Concrete fix to evaluate (needs the storage-mode owner + a restart e2e — do NOT ship blind):

• Give node-local durable storage by mounting the k3s local-path storage dir to the host: k3d cluster create … --volume <host>:/var/lib/rancher/k3s/storage@server:0 (+ the ps1 equivalent), so PVs survive a node-container recreation. Verify against _check_existing_cluster_storage_mode and both OS paths.
• Add a node-restart → re-provision cycle to the cluster/auto-upgrade e2e (needs a real WSL/Docker restart — ties to feat(ci): real Windows e2e leg for the installer — scheduled VM with nested virtualization (RFC-CLIENT-0003 D5) #436's runner).
• Interim safety: ensure the install-time warning that node-local data is non-durable is surfaced to the operator (partial coverage exists at cluster.sh:713 / :377).

Deferring the code change to the storage-mode owner (@saadqbal / #456) since it changes storage topology and can't be validated without a restart-cycle e2e.