feat: OpenPlotter → Halos in-place migration scripts + runbook

[mairas]

Overview

Two CLI bash scripts plus a user-facing runbook that let an OpenPlotter (standard Raspberry Pi OS) marine user migrate their data to a freshly-flashed marine Halos install on the same device, in place, using an external USB stick as transport. openplotter-backup.sh runs on the live OpenPlotter system and copies Signal K, InfluxDB 2, Grafana, and OpenCPN data to the stick; the user reflashes to Halos (existing docs); halos-restore.sh runs on the fresh Halos system and reinstates the data, aligning it to Halos' marine defaults. Generalized from a real one-off migration.

Problem Frame

OpenPlotter users adopting Halos lose years of marine telemetry, Signal K configuration, Grafana dashboards, and OpenCPN setup unless they can carry it across. The original migration was done by hand across three hosts with 200+ ad-hoc commands — too error-prone and host-specific for a customer. This packages the validated procedure into two scripts safe enough for a non-expert, plus a runbook. Defining constraint: the reflash is in place and irreversible, so during the wipe window the USB stick holds the only copy of the data — backup must prove completeness before the user reflashes.

Requirements Trace

• R1. Back up Signal K, InfluxDB 2, Grafana, and OpenCPN data from a live OpenPlotter system to a USB stick.
• R2. Restore that data into a fresh marine Halos install so historical telemetry, dashboards, SK config, and OpenCPN setup are preserved and queryable.
• R3. Backup must verify completeness and present an explicit "safe to reflash" gate before the user wipes.
• R4. Backup must run on foreign territory with only base tooling (coreutils, tar, systemctl) — no docker, sqlite3, or influx CLI assumed on OpenPlotter.
• R5. Restore must align boat data to Halos marine defaults so Halos' own provisioned Grafana datasource works against the historical data, and ongoing Signal K logging continues into the right bucket.
• R6. Scripts delivered by fetch-then-run from stable docs.halos.fi/migrate/ URLs; runbook published in the docs site beside the existing migration guide.
• R7. CLI only. No packaging, no new repo, no GUI.

Scope Boundaries

• No HALPI2 / Halos flashing or reflashing instructions — link existing docs.
• No Debian package, no new repo, no Cockpit/web/GUI.
• No automated migration of ~/docker app data or /etc system config (boat-specific; out of scope).
• No cross-device migration; single-device, in-place only.
• Not swapping the Grafana data directory — dashboards/datasources are re-imported via the Grafana API into Halos' own Grafana (OIDC/provisioning intact).
• Charts in user-relocated OpenCPN folders are best-effort: detected and warned, not guaranteed.

Context & Research

Target: marine Halos (halos-marine-containers)

• InfluxDB — apps/influxdb/: service marine-influxdb-container.service, container influxdb, image influxdb:2.9.1. Raw data dir /var/lib/container-apps/marine-influxdb-container/data/db → container /var/lib/influxdb2. Env /etc/container-apps/marine-influxdb-container/env, token var INFLUXDB_ADMIN_TOKEN. Defaults: org marine, bucket marine, admin pw halos-default, token placeholder halos-default-token-change-in-production. prestart regenerates the token only if placeholder/empty; also runs a password-sync step (verify it doesn't disturb an injected token).
• Grafana — apps/grafana/: service marine-grafana-container.service, container grafana, image grafana/grafana:13.0.1, port 3000 on halos-proxy-network. Provisioned datasource: name InfluxDB, db marine, token via $__env{INFLUXDB_TOKEN}, isDefault: true, auto UID. Auth is OIDC/Authelia; no admin password set → built-in admin:admin is what the original run used against the internal IP (verify).
• Signal K — apps/signalk-server/: service marine-signalk-server-container.service, container signalk-server, image signalk-server:v2.27.0. Data dir /var/lib/container-apps/marine-signalk-server-container/data/data → /home/node/.signalk. prestart creates security.json only if missing, and auto-configures signalk-to-influxdb2 for org/bucket marine.

Source: OpenPlotter / standard Raspberry Pi OS

• Signal K: ~/.signalk (exclude node_modules, skserver-raw_*.log); service likely signalk.service (confirm).
• InfluxDB 2: /var/lib/influxdb2, service influxdb.service.
• Grafana: /var/lib/grafana/grafana.db, service grafana-server.service.
• OpenCPN: config ~/.opencpn, plugin data ~/.local/share/opencpn; charts in user-defined dirs (relocate risk). Desktop app, no service — must be closed during backup.

Docs site

MkDocs Material, site_url: https://docs.halos.fi, GitHub Pages via .github/workflows/deploy.yml (mkdocs build --strict). Existing guide docs/user-guide/app-data.md. Scripts serve at docs.halos.fi/migrate/... (not halos.fi/..., not served by this repo).

Key Technical Decisions

• Raw InfluxDB data-dir swap, not portable backup/restore — portable path was tried and abandoned; raw swap is proven.
• Mint the operator token at restore time from the boat's influxd.bolt via influxd recovery auth create-operator (through docker run --rm influxdb:2.9.1). No source credentials needed.
• Rename the boat's data-bearing org + bucket to marine. Source names are arbitrary and boat-specific (OpenPlotter imposes no default; hurma/skdata were just this boat's). Discover by id, identify the telemetry bucket by data presence, and prompt the user when more than one candidate exists — never hardcode. The raw swap carries all orgs/buckets verbatim; the rename only retargets the primary one so Halos' default datasource and the SK influx writer line up. Additional buckets remain present under original names.
• Add v1 DBRP marine/autogen (default) for InfluxQL/Grafana-v1; the boat's legacy skdata-db/skdata-rp DBRPs ride along in the raw data and back imported boat dashboards via a recreated boat datasource with preserved UID.
• Grafana dashboards/datasources re-imported via API, Grafana data dir untouched. Restore reads the boat grafana.db with python3 stdlib sqlite3 (no host sqlite3 dependency).
• Backup stops source services for a consistent raw copy, then restarts them.
• USB is a genuine cross-filesystem copy both ways — no same-filesystem mv shortcut.
• Manifest with format_version written by backup, validated by restore; refuses on mismatch; records per-domain presence/sizes for the completeness gate.
• Halos defaults preserved aside before any swap: db.halos-default, env.halos-default, data.halos-default (recovery = rename back).

Open Questions

Resolved during planning

• Scripts URL → docs.halos.fi/migrate/*.sh.
• Renaming to marine integrates with Halos defaults (org+bucket default is marine).
• Read grafana.db on Halos without extra deps → python3 stdlib sqlite3.
• Same-filesystem move optimization → not applicable across USB.
• Additional (non-telemetry) buckets → no special handling; raw swap carries all buckets verbatim; rename retargets only the primary telemetry bucket; prompt is solely about choosing the rename target.

Deferred to implementation

• Exact OpenPlotter Signal K service name(s) — detect dynamically; tolerate absence.
• Whether Grafana built-in admin:admin still authenticates given OIDC — verify; fall back to a temp admin.
• Whether influx prestart password-sync disturbs the injected operator token — verify token survives restart.
• Reconciling the restored SK influx-writer plugin config (arbitrary boat org/bucket) with the renamed marine bucket and Halos prestart auto-config — rewrite, defer to prestart, or document a manual step.
• Whether mkdocs build --strict warns on raw .sh under docs/ (nav-exclude vs docs/assets/).
• USB "clearly too small" threshold heuristic.

High-Level Technical Design

Directional guidance for review, not implementation specification.

sequenceDiagram
    participant U as User
    participant OP as OpenPlotter (live)
    participant USB as USB stick
    participant HA as Halos (fresh)
    U->>OP: fetch + run openplotter-backup.sh
    OP->>OP: stop signalk/influxdb/grafana, close OpenCPN
    OP->>USB: copy ~/.signalk, /var/lib/influxdb2,<br/>grafana.db, ~/.opencpn (+manifest)
    OP->>OP: verify completeness, restart services
    OP-->>U: "SAFE TO REFLASH"
    U->>HA: reflash to Halos (existing docs), then fetch + run halos-restore.sh
    HA->>USB: validate manifest (format_version)
    HA->>HA: SK: default aside, restore data, start, verify API
    HA->>HA: Influx: mint token (bolt), swap db, inject token,<br/>rename org/bucket→marine, DBRP, smoke query
    HA->>HA: Grafana: read boat grafana.db (python3),<br/>recreate boat DS (preserved UID), import dashboards
    HA->>HA: OpenCPN: restore ~/.opencpn + ~/.local/share/opencpn
    HA-->>U: health check + verification summary

Loading

Risks & Dependencies

Risk	Mitigation
USB is the only data copy during the irreversible wipe	Hard completeness gate; "safe to reflash" only on verified backup; runbook leads with this
Boat InfluxDB version newer than Halos influxd (2.9.1) → raw data won't open	Document precondition; restore surfaces start failure, keeps db.halos-default for rollback
influx prestart password-sync clobbers injected operator token	Inject token before first start; verify it survives restart
Source org/bucket names arbitrary; multiple buckets may exist	Discover by id, identify telemetry bucket by data presence, prompt when ambiguous; never hardcode
Ongoing SK logging writes to old bucket name after migration	Rename to marine + reconcile SK influx plugin config; documented manual step as fallback
Grafana OIDC blocks admin:admin API access	Verify; fall back to provisioning a temporary admin
OpenCPN charts in relocated user folders not captured	Detect from opencpn.conf, warn; document as best-effort
mkdocs --strict rejects raw .sh in docs tree	Confirm/adjust placement or nav config

Implementation Units

• Unit 1: openplotter-backup.sh — OpenPlotter-side collector #25 — Unit 1: openplotter-backup.sh (OpenPlotter-side collector)
• Unit 2: halos-restore.sh — Signal K + OpenCPN restore #26 — Unit 2: halos-restore.sh (Signal K + OpenCPN restore)
• Unit 3: halos-restore.sh — InfluxDB + Grafana restore #27 — Unit 3: halos-restore.sh (InfluxDB + Grafana restore)
• Unit 4: Migration runbook page #28 — Unit 4: Migration runbook page
• Unit 5: Static script serving + build wiring #29 — Unit 5: Static script serving + build wiring

[mairas]

Implementation status — all five units landed on feat/openplotter-migration-scripts

Commits: backup script, restore (SK+OpenCPN), restore (InfluxDB+Grafana), runbook + nav/serving.

Deferred questions resolved during implementation (verified on a live HaLOS device, halosdev):

• Grafana admin:admin works via the container's internal IP despite OIDC — no temp-admin fallback needed in practice.
• The injected INFLUXDB_ADMIN_TOKEN survives a service restart — influx prestart's password-sync does not clobber it.
• InfluxDB data/db ownership is pi:root mode 700; restore mirrors whatever HaLOS sets on the default dir.
• Operator-token mint from the boltdb authenticates fine even with use-hashed-tokens: true (the hashing warning is benign). create-operator requires an existing username, so restore discovers it via influxd recovery user list.
• Historical data verified queryable in the renamed marine bucket via both Flux and InfluxQL v1 (the marine/autogen DBRP works).

New limitation found (documented as best-effort in the runbook):

• Grafana 13+ no longer stores dashboards in dashboard.data (unified storage), so dashboard import depends on the source Grafana version. Datasources still import; the importer degrades gracefully and HaLOS ships its own marine dashboards regardless.

Real-world wrinkle handled beyond the plan: cheap USB sticks are often FAT32 (4 GB file cap, no permission preservation), so the backup tars each domain (ownership preserved across filesystems) and refuses early on FAT32 when InfluxDB data exceeds 4 GB.

Still unverifiable without OpenPlotter hardware: the exact Signal K service name on OpenPlotter (backup tolerates absence and stops any signalk* unit it finds).