AIL-262: feat(edge): collect launchpad-ai + GPU + storage state for AI-appliance triage

[rishi-anand]

Jira: AIL-262

Summary

Extends support-bundle-edge.sh to capture the host + cluster state needed to root-cause failures on Palette AI-inference appliances (launchpad-ai on MI325X / H200 / B200), and to close storage/LVM/NVMe gaps that affect any edge host. Every command added is read-only and fails gracefully if its dependency isn't installed or its target isn't present.

What's added

1. AI namespaces to SYSTEM_NAMESPACES

launchpad-ai, amd-gpu-operator, gpu-operator. Their pods, previous-instance logs, and CRs get pulled by the existing generic namespace + CRD walks. All launchpad.spectrocloud.com CRDs (models, launchpadconfigs, modelgroupquotas, modelgroupapikeys, quotasettings) are covered automatically.

2. Previous boot's kernel log

journald-log() now also collects dmesg-previous-boot and journalctl-previous-boot when persistent journald is enabled. Critical for post-crash forensics — GPU wedges typically freeze host networking and force a reboot, so the current-boot dmesg after such a reboot is post-recovery and useless.

3. New storage-info() — always safe to run

Read-only host storage collection, every tool gated on command -v, timeout-wrapped where a wedged controller could hang:

• lsblk -f, lsblk -o … (fs/UUID/model/serial/rotational)
• df -h, df -i, /proc/mounts, /etc/fstab
• fdisk -l
• LVM: pvdisplay, vgdisplay, lvdisplay, pvs, vgs, lvs
• NVMe: nvme list, nvme id-ctrl, nvme smart-log per drive (10-15s each)
• SMART: smartctl -a per block device (20s each; no -t — never triggers tests)
• Sysfs block attributes (size, hidden, rotational, dm/uuid, dm/name, device/{model,vendor,serial}) — mirrors what stylus/pkg/disk reads for enumeration
• /dev/disk/by-* full symlink tree (covers by-uuid, by-partuuid, by-label, by-partlabel, by-path, by-id)
• dmsetup ls --tree, dmsetup info — device-mapper topology for LVM / multipath / dm-crypt
• /sys/class/nvme-subsystem/ — native-NVMe-multipath subsystem tree

4. New gpu-info() — hardware-vendor gated

Emits gpu/ directory. AMD block gated on [ -d /sys/module/amdgpu ]; NVIDIA block gated on [ -e /proc/driver/nvidia ] || command -v nvidia-smi. Emits an empty gpu/ dir on non-GPU hosts.

AMD (when amdgpu present):

• Module version + modinfo + /sys/module/amdgpu/parameters/* (lockup_timeout, reset_method, gpu_recovery)
• Per-card VBIOS via /sys/bus/pci/devices/*/vbios_version
• Per-card firmware components (sdma_fw_version, sos_fw_version, smc_fw_version, ta_xgmi_fw_version, mec_fw_version, rlc_fw_version, …)
• Per-card RAS + ECC counters via /sys/bus/pci/devices/*/ras/*
• Per-card temp/power/voltage/fan via hwmon
• rocm-smi -a and rocm-smi --show{hw,driverversion,vbios,temp,use,power,fw,bios} — from host if installed, else via kubectl exec into an operator/vLLM pod (timeout-guarded)
• AMD GPU devcoredumps from /var/log/amdgpu-devcoredump/*.bin (populated by AIL-263 / launchpad-ai#512)
• Live /sys/class/devcoredump/ listing (no reads of data — reads would consume)

NVIDIA (when nvidia driver present):

• /proc/driver/nvidia/version
• nvidia-smi -q, nvidia-smi topo -m, compact query CSV (pstate/temp/util/memory/ECC-corrected/ECC-uncorrected)
• nvidia-bug-report.sh if installed (comprehensive vendor dump; 180s hard cap)
• NVIDIA GPU devcoredumps from /var/log/nvidia-devcoredump/*.bin (populated by AIL-264 / launchpad-ai#530)

launchpad-ai on-disk (both vendors):

• Directory shape of /opt/data/spectrocloud/models/ — weights are hundreds of GB, never copied into the bundle
• Every model's metadata.yaml (~20 KB each; declares the launch recipe)

Read-only + graceful-fail guarantee

Read-only. Verified by grepping for --set, --reset, smartctl -t, nvme format, dmsetup remove/create/load, pvcreate/pvremove, vgcreate/vgremove, lvcreate/lvremove/lvresize, parted mkpart. Zero matches. Every GPU/storage tool uses only --show* / --query* / -l / list / display / info / id-ctrl / smart-log flags.

Graceful-fail.

• Every optional binary gated on command -v — silent skip if not installed
• Every vendor block gated on hardware presence (/sys/module/amdgpu, /proc/driver/nvidia) — no-op on non-GPU nodes
• Every command that could hang wrapped in timeout N: nvme (10-15s), smartctl (20s), kubectl exec (10-30s), nvidia-bug-report.sh (180s hard cap)
• Script does NOT use set -e (verified — existing pattern), so individual command failures don't abort the bundle
• 13 timeout wrappers in the new code; 61 gated conditionals total

Motivation

Grounded in the July 2026 MI325X incident (see AIL-262) where two vLLM crashes each wedged the host. We had to log in mid-triage and hand-run every one of these commands. This PR turns the ad-hoc triage checklist into a first-class collector so support engineers don't repeat it.

Companion PRs that install the udev rules this collector reads:

• AIL-263 / launchpad-ai#512 (AMD)
• AIL-264 / launchpad-ai#530 (NVIDIA)

Diff size

support-bundle/README-edge.md         |  49 ++
support-bundle/support-bundle-edge.sh | 258 ++++++++++++++
2 files changed, ~307 insertions

Test plan

• Run on the MI325X host (137.220.60.220): confirm gpu/amd/ gets rocm-smi + per-card firmware/VBIOS + RAS state; storage/ gets pvdisplay + lsblk + nvme SMART.
• Run on an H200 / B200 host: confirm gpu/nvidia/ gets nvidia-smi + optional nvidia-bug-report.
• Run on a non-GPU edge host: confirm gpu/ is empty and everything else is unaffected. ← critical graceful-fail test
• Run on a minimal Kairos host without nvme-cli or smartmontools: confirm storage/ still gets the sysfs + lsblk output and skips the missing tools without erroring. ← critical graceful-fail test
• bash -n support-bundle-edge.sh — passes ✅
• Introduce a synthetic hang in an exec target (e.g. unresponsive pod) and confirm the bundle completes within the timeout bounds.

🤖 Generated with Claude Code