swares · swares · Jul 18, 2026 · Jul 18, 2026 · Jul 18, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -1,8 +1,8 @@
 # CLAUDE.md — operating rules for this lab
 
-You are helping operate a home lab whose core is an **Odroid-H4 Ultra** that is *both* a
-NAS and a single-node **k3s** cluster (Ubuntu 22.04). Read this before acting. Full context
-is in `docs/` (start with `ARCHITECTURE.md`).
+You are helping operate a home lab whose core is an **Odroid-H4 Ultra** NAS, part of a
+**3-node HA k3s cluster** (H4 + n150-1 + n150-2 as servers; opi5pro-1/2 as ARM64 agents).
+Read this before acting. Full context is in `docs/` (start with `ARCHITECTURE.md`).
 
 ## How changes are made
 
@@ -65,7 +65,7 @@ is in `docs/` (start with `ARCHITECTURE.md`).
 
 The H4 is the core (k3s server + NAS, Ubuntu 22.04, `192.168.1.160`). The two
 **Orange Pi 5 Pro** boards (8C/16GB/NPU) are k3s agents / AI inference hosts; RPi 5
-runs Vault; RPi 4B runs Pi-hole (secondary DNS, 192.168.1.116); lldap runs as a k3s Deployment in the `lldap` namespace (ldap-1 VM decommissioned
+runs Vault; RPi 4B runs Pi-hole (secondary DNS, 192.168.1.116); Home Assistant runs as a k3s Deployment in the `home-assistant` namespace; lldap runs as a k3s Deployment in the `lldap` namespace (ldap-1 VM decommissioned
 2026-07-04); the XU3 is a build agent. DNS needs a permanent host.
 M5Stack + OPi NPUs are edge inference endpoints, not cluster nodes.
 The map's plaintext credentials must be rotated. See `docs/HARDWARE.md`.
diff --git a/README.md b/README.md
@@ -1,10 +1,10 @@
 # Home Lab — DevOps Environment
 
-A GitOps-managed home lab built around an **Odroid-H4 Ultra** that serves double duty as
-a NAS and a single-node **k3s** cluster, with an ARM inference fleet. Infrastructure is
-defined as code: **Ansible** stands hosts up, **Argo CD** runs everything inside the
-cluster from this git repo, and a two-tier storage model (hot NVMe + two cold SATA RAID 1
-mirrors) keeps data safe.
+A GitOps-managed home lab built around an **Odroid-H4 Ultra** NAS core and a
+**3-node HA k3s cluster** (H4 + two N150 mini PCs), with an ARM64 inference fleet
+(two Orange Pi 5 Pros). Infrastructure is defined as code: **Ansible** stands hosts up,
+**Argo CD** runs everything inside the cluster from this git repo, and a two-tier storage
+model (hot NVMe + two cold SATA RAID 1 mirrors) keeps data safe.
 
 ## Why it's shaped this way
 
@@ -34,47 +34,71 @@ mirrors) keeps data safe.
 
 | App | Namespace | Notes |
 |-----|-----------|-------|
-| Immich | `immich` | Photo server + Postgres + Redis + ML; library on cold RAID at `/mnt/cold-8t/immich` |
-| LiteLLM gateway | `ai-gateway` | Load-balances `chat` across both Ollama instances (simple-shuffle) |
-| Ollama | `ai-gateway` | qwen2.5:3b on opi5pro-1, pinned via nodeSelector |
-| Ollama-2 | `ai-gateway` | qwen2.5:3b on opi5pro-2, pinned via nodeSelector |
-| kube-prometheus-stack | `monitoring` | Prometheus (30d/50GB), Grafana, Alertmanager, node-exporter on all nodes |
+| Immich | `immich` | Photo server + Postgres (vectorchord) + Redis + ML; library on NFS ReadWriteMany PV |
+| LiteLLM gateway | `ai-gateway` | Unified OpenAI-compatible API (`ai.apps.lab.home.arpa`) across all backends |
+| RKLLama | `ai-gateway` | NPU-native LLM on opi5pro-1/2 (DeepSeek-R1-Distill-Qwen-1.5B, ~7–8 tok/s) |
+| Ollama | `ai-gateway` | In-cluster fallback engine on opi5pro-1/2; pinned to `ollama/ollama:0.32.0` |
+| m5stack-adapter | `ai-gateway` | OpenAI shim for M5Stack `/api/*` protocol; image 0.1.1 |
+| Whisper STT | `whisper` | Speech-to-text at `https://stt.apps.lab.home.arpa`; CPU on n150-1 |
+| lldap | `lldap` | Lightweight LDAP directory; web UI at `lldap.apps.lab.home.arpa` |
+| Authelia | `authelia` | OIDC/SSO backed by lldap; PostgreSQL backend; `authelia.apps.lab.home.arpa` |
+| Home Assistant | `home-assistant` | `ha.apps.lab.home.arpa`; MQTT consumer (broker at opi-zero2w-2 .188) |
+| Minio | `minio` | S3-compatible object store; `tofu-state` bucket holds OpenTofu state |
+| Semaphore | `semaphore` | Ansible UI at `semaphore.apps.lab.home.arpa` |
+| Kyverno | `kyverno` | 3 ClusterPolicies in Enforce mode (no-latest-tag, resource-limits, no-privileged) |
+| kube-prometheus-stack | `monitoring` | Prometheus (30d/40GB), Grafana, Alertmanager, Loki, Alloy on all nodes |
 | external-secrets | `external-secrets` | Pulls secrets from Vault (KV v2 at `secret/lab/`) |
-| cert-manager | `cert-manager` | TLS certificate management |
-| Argo CD | `argocd` | GitOps controller — manages all of the above |
+| cert-manager | `cert-manager` | TLS via `lab-ca` ClusterIssuer (self-signed root CA) |
+| Argo CD | `argocd` | GitOps controller — selfHeal + prune on all apps |
 
 ### Host services (outside k3s)
 
 | Service | Host | IP |
 |---------|------|----|
-| HashiCorp Vault v1.21.0 | rpi5 | 192.168.1.128 |
-| Mosquitto MQTT | opi-zero2w-2 | 192.168.1.188 |
-| Pi-hole DNS (primary) | octopi | 192.168.1.148 |
+| HashiCorp Vault | rpi5 | 192.168.1.128 |
+| Mosquitto MQTT (primary) | opi-zero2w-2 | 192.168.1.188 |
+| Mosquitto MQTT (secondary, HA bridge) | opi-zero2w-4 | 192.168.1.99 |
+| Pi-hole DNS (primary, v6.4.3) | octopi (RPi 3B #2) | 192.168.1.148 |
+| Pi-hole DNS (secondary, v6) | rpi4b (RPi 4B) | 192.168.1.116 |
+| dnsmasq DNS (tertiary fallback) | opi-zero2w-1 | 192.168.1.184 |
 | Samba / NFS (NAS) | H4 (host) | 192.168.1.160 |
+| GitLab CE | gitlab-1 VM on n150-1 | 192.168.1.50 |
 
 ### Ingress endpoints
 
 | URL | Service |
 |-----|---------|
-| `grafana.apps.lab.home.arpa` | Grafana |
 | `argocd.apps.lab.home.arpa` | Argo CD |
+| `grafana.apps.lab.home.arpa` | Grafana |
 | `immich.apps.lab.home.arpa` | Immich |
-| `*.apps.lab.home.arpa` | → 192.168.1.160 (Traefik) |
+| `ai.apps.lab.home.arpa` | LiteLLM gateway |
+| `ha.apps.lab.home.arpa` | Home Assistant |
+| `authelia.apps.lab.home.arpa` | Authelia (SSO) |
+| `lldap.apps.lab.home.arpa` | lldap (LDAP directory UI) |
+| `semaphore.apps.lab.home.arpa` | Semaphore (Ansible UI) |
+| `minio.apps.lab.home.arpa` | Minio (S3 API) |
+| `minio-console.apps.lab.home.arpa` | Minio console |
+| `stt.apps.lab.home.arpa` | Whisper STT |
+| `*.apps.lab.home.arpa` | → 192.168.1.160 (Traefik ingress) |
 
 ## Fleet
 
 | Host | Ansible name | IP | Role |
 |------|--------------|----|------|
-| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS |
-| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, Ollama inference |
-| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, Ollama-2 inference |
-| Raspberry Pi 5 | rpi5 | 192.168.1.128 | Vault |
-| N150 mini PC #1 | n150-1 | 192.168.1.42  | Ubuntu 24.04, k3s server + KVM hypervisor |
-| N150 mini PC #2 | n150-2 | 192.168.1.21  | Ubuntu 24.04, k3s server + KVM hypervisor |
-| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed, sleep disabled) |
-| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT broker |
-| OctoPi (RPi 3B #2) | octopi | 192.168.1.148 | Pi-hole DNS (Bookworm, Pi-hole v6.4.3) |
-| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent (flagged unstable) |
+| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS (smbd/NFS) |
+| N150 mini PC #1 | n150-1 | 192.168.1.42 | k3s server + KVM hypervisor (Ubuntu 24.04) |
+| N150 mini PC #2 | n150-2 | 192.168.1.21 | k3s server + KVM hypervisor (Ubuntu 24.04) |
+| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, RKLLama/Ollama inference, NPU |
+| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, RKLLama/Ollama inference, NPU |
+| Raspberry Pi 5 | rpi5 | 192.168.1.128 | HashiCorp Vault |
+| Raspberry Pi 4B | rpi4b | 192.168.1.116 | Pi-hole secondary DNS (v6, Bookworm) |
+| RPi 3B #2 (octopi) | octopi | 192.168.1.148 | Pi-hole primary DNS (v6.4.3, Bookworm) |
+| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed) |
+| OPi Zero 2W #1 | opi-zero2w-1 | 192.168.1.184 | dnsmasq DNS tertiary fallback |
+| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT primary broker |
+| OPi Zero 2W #3 | opi-zero2w-3 | 192.168.1.217 | dnsmasq DNS secondary (Armbian Trixie) |
+| OPi Zero 2W #4 | opi-zero2w-4 | 192.168.1.99 | MQTT secondary broker (HA bridge) |
+| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent |
 
 ## Quickstart (fresh bootstrap)
 
@@ -134,13 +158,29 @@ See [docs/SECURITY.md](docs/SECURITY.md) for the full security model.
 ## TODO
 
 ### Actionable now
-- [ ] Rotate octopi Pi-hole credentials (currently plaintext)
-- [ ] Flash octopi to Raspberry Pi OS Bookworm → upgrade to Pi-hole v6
-- [ ] Move photo library (~1.3TB) into `/mnt/cold-8t/immich` so Immich can serve it
+- [ ] Rotate sudo passwords on n150-1/n150-2 (exposed in terminal output 2026-07-18 — run `rotate-passwords.yml`)
+- [ ] Upgrade rknpu driver 0.9.6 → 0.9.7 to unlock 3B+ model support on opi5pro-1/2
+- [ ] Fix Ollama image pin + Whisper versioned tag (required by Kyverno `disallow-latest-tag` policy)
+- [ ] Investigate Authelia health stuck Progressing in ArgoCD
 
 ### Longer term
 - [ ] Vault TLS — currently plain HTTP; wire cert-manager before exposing beyond LAN
-- [ ] LDAP integration with Immich SSO once OpenLDAP is up
-- [ ] n150-1/2 → OVMS Intel iGPU for embeddings (`embeddings` model in LiteLLM config already wired)
-- [ ] Offsite restic backup target
-- [ ] Pin Immich image tag (currently `:release` floating) — Renovate will open the first PR
+- [ ] Windows Update automation (`windows-updates.yml`) — `ansible.windows.win_updates` module ready, playbook not yet written
+- [ ] Remote access — WireGuard or Tailscale
+
+### Done ✅
+- [x] octopi flashed to Bookworm, Pi-hole v6.4.3 running (2026-07-13)
+- [x] RPi 4B: Pi-hole v6 secondary DNS live at 192.168.1.116 (2026-07-02)
+- [x] n150-1/n150-2 joined as k3s server nodes, kube-vip VIP 192.168.1.200 (2026-07-02)
+- [x] lldap migrated from ldap-1 VM to k3s Deployment in `lldap` namespace (2026-07-04)
+- [x] Authelia → PostgreSQL backend (2026-07-03)
+- [x] Immich library → NFS ReadWriteMany PV (schedules on any node) (2026-07-04)
+- [x] Shared NFS storage between n150-1/n150-2 for VM live migration (2026-07-03)
+- [x] Monitoring stack migrated to n150-1 (2026-07-04)
+- [x] zswap on n150-1/n150-2 (zstd, zsmalloc, 20%) (2026-07-03)
+- [x] MQTT HA: opi-zero2w-4 secondary broker with bidirectional bridge (2026-07-10)
+- [x] Kyverno 3 ClusterPolicies in Enforce mode (2026-07-14)
+- [x] ArgoCD notifications + git-directory ApplicationSet (2026-07-14)
+- [x] Semaphore Ansible UI live (2026-07-18)
+- [x] OpenTofu state in Minio; gitlab-1 VM codified (2026-07-18)
+- [x] Offsite restic backup target (backup-offsite.timer)
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
@@ -133,13 +133,14 @@ The lab is a flat **192.168.1.0/24** network. **DNS is the linchpin of the insta
 k3s needs `api.lab.home.arpa → 192.168.1.200` (kube-vip VIP) and
 `*.apps.lab.home.arpa → 192.168.1.160` (H4 Traefik ingress).
 
-Three DNS servers provide redundancy:
+Four DNS servers provide redundancy:
 
 | Role | Host | IP | Engine |
 |------|------|----|--------|
-| Primary | RPi 3B #2 (octopi) | `192.168.1.148` | Pi-hole (Bookworm pending) |
-| Secondary | RPi 4B | `192.168.1.116` | Pi-hole v6 |
+| Primary | RPi 3B #2 (octopi) | `192.168.1.148` | Pi-hole v6.4.3 (Bookworm, confirmed 2026-07-13) |
+| Secondary | RPi 4B | `192.168.1.116` | Pi-hole v6 (Bookworm) |
 | Tertiary | OPi Zero 2W #1 | `192.168.1.184` | dnsmasq (fallback) |
+| Quaternary | OPi Zero 2W #3 | `192.168.1.217` | dnsmasq (Armbian Trixie, 2026-07-10) |
 
 Inside the cluster, CoreDNS has a custom zone (`coredns-custom` ConfigMap in kube-system)
 that answers all `*.apps.lab.home.arpa` queries with `192.168.1.160`, ensuring pods on any
@@ -198,3 +199,20 @@ Grouped services and the hosts that provide them:
 An **interactive, filterable** version (toggle hardware classes, select-all/clear) is at
 [service-map.html](service-map.html) — open it in a browser. A **host-centric** companion
 (one card per box, everything it runs, same filters) is at [host-map.html](host-map.html).
+
+### Key cluster services (as of 2026-07-18)
+
+| Service | Namespace | Notes |
+|---------|-----------|-------|
+| Argo CD | `argocd` | GitOps; app-of-apps + git-directory ApplicationSet |
+| Immich | `immich` | Photos/video; NFS ReadWriteMany library PV |
+| Home Assistant | `home-assistant` | HA automation; MQTT consumer |
+| lldap | `lldap` | Lightweight LDAP; migrated from ldap-1 VM (2026-07-04) |
+| Authelia | `authelia` | OIDC/SSO; PostgreSQL backend |
+| LiteLLM + AI backends | `ai-gateway` | RKLLama (NPU), Ollama (fallback), m5stack-adapter, Whisper |
+| Semaphore | `semaphore` | Ansible UI; runs playbooks from inside the cluster |
+| Minio | `minio` | S3 object store; holds OpenTofu state |
+| Kyverno | `kyverno` | Policy enforcement (3 ClusterPolicies, Enforce mode) |
+| kube-prometheus-stack | `monitoring` | Prometheus + Grafana + Alertmanager + Loki + Alloy |
+| External Secrets | `external-secrets` | Syncs Vault KV v2 → k8s Secrets |
+| cert-manager | `cert-manager` | `lab-ca` ClusterIssuer; signs `*.apps.lab.home.arpa` TLS |
diff --git a/docs/HARDWARE.md b/docs/HARDWARE.md
@@ -1,7 +1,7 @@
 # Hardware Inventory & Roles
 
 Subnet: `192.168.1.0/24`. Gateway: `192.168.1.1`.
-Last verified: 2026-07-10.
+Last verified: 2026-07-18.
 
 > **Security:** Default passwords (`pi`/`odroid`/`root`) have been rotated on all managed
 > hosts via `ansible/playbooks/rotate-passwords.yml`. Credentials are in Ansible Vault —
@@ -18,7 +18,7 @@ Last verified: 2026-07-10.
 | **N150 mini PC #2** | Intel N150 4C x86 | 16 GB | `192.168.1.21` (br0) | — | KVM hypervisor · Ubuntu 24.04 |
 | **N150 mini PC #3 (HTPC)** | Intel N150 4C x86 | 16 GB | — | `192.168.1.176` | Living-room HTPC |
 | **RPi 5** | Cortex-A76 4C | 8 GB | `192.168.1.128` | `192.168.1.124` (avoid) | HashiCorp Vault |
-| **RPi 4B** | Cortex-A72 4C | 8 GB | `192.168.1.116` | — | Home Assistant host (Debian 12 Bookworm) |
+| **RPi 4B** | Cortex-A72 4C | 8 GB | `192.168.1.116` | — | Pi-hole secondary DNS (Debian 12 Bookworm) |
 | **RPi 3B #2** | Cortex-A53 4C | 1 GB | `192.168.1.148` | `192.168.1.152` (avoid) | DNS primary (Pi-hole v6.4.3, Bookworm) |
 | **RPi 3B #1** | Cortex-A53 4C | 1 GB | — | — | ⚠ on-board power fault — retired |
 | **Odroid-XU3 #1** | Exynos5422 8C | 2 GB | `192.168.1.64` | — | Build agent (Python <3.8 — excluded from Ansible auto-updates) |
@@ -33,6 +33,7 @@ Last verified: 2026-07-10.
 
 | VM | Host | IP | OS | Role |
 |----|------|----|----|------|
+| **gitlab-1** | n150-1 | `192.168.1.50` | Ubuntu 22.04 | GitLab CE Omnibus · `gitlab.lab.home.arpa` · 80 GiB qcow2 at `/var/lib/libvirt/images/gitlab-1.qcow2` · MAC `52:54:00:6b:ab:01` · UUID `6ea193a5-61f9-4b65-8ee1-a90b343aec5f` · codified in `tofu/vms/` |
 | **ldap-1** | n150-1 | `192.168.1.70` | Ubuntu 24.04 | ~~lldap v0.6.3~~ **decommissioned 2026-07-04** — lldap migrated to k3s `lldap` namespace |
 
 ## Network
@@ -130,10 +131,18 @@ CoreDNS extended with `coredns-custom` ConfigMap for in-cluster `*.apps.lab.home
 - ✅ n150-1/n150-2 sudo password: Ansible-Vault-encrypted in group_vars/kvm_hosts/secrets.yml; sync-secrets-to-vault.yml now backs it up to secret/lab/hosts (2026-07-03)
 - ✅ ArgoCD deploy key: sync-secrets-to-vault.yml backs up ~/.ssh/argocd-deploy-key to secret/lab/argocd (2026-07-03)
 - ✅ zswap enabled on n150-1/n150-2: zstd compressor, zsmalloc zpool, 20% max pool (2026-07-03)
-- N150 #3: WinRM credentials (wrong password on file)
-- ✅ Authelia → PostgreSQL: postgres.yaml added, configmap+deployment updated, replicas:2 — pending password generation + git push (2026-07-03)
+- ✅ N150 #3: WinRM credentials corrected (2026-07-13)
+- ✅ Authelia → PostgreSQL: postgres.yaml added, configmap+deployment updated (2026-07-03)
 - ✅ Shared NFS storage between n150-1/n150-2: /srv/libvirt-shared, libvirt-shared pool active on both nodes, SSH key exchange complete — VM live migration ready (2026-07-03)
 - ✅ Monitoring stack (Prometheus, Grafana, Alertmanager, Loki) migrated from odroid-nas to n150-1 (2026-07-04)
 - ✅ immich-library PV: hostPath+nodeAffinity → NFS ReadWriteMany; immich-server now schedules on any node (2026-07-04)
-- ✅ lldap migrated from ldap-1 KVM VM (192.168.1.70) to k3s Deployment in lldap namespace; ldap-1 VM stopped (2026-07-04)
-- ✅ MQTT HA: opi-zero2w-4 (.99) deployed as secondary broker bridging all topics to/from opi-zero2w-2 (.188); M5Stack firmware has automatic failover (2
+- ✅ lldap migrated from ldap-1 KVM VM (192.168.1.70) to k3s Deployment in lldap namespace; ldap-1 VM decommissioned (2026-07-04)
+- ✅ MQTT HA: opi-zero2w-4 (.99) deployed as secondary broker bridging all topics to/from opi-zero2w-2 (.188); M5Stack firmware has automatic failover (2026-07-10)
+- ✅ Kyverno: 3 ClusterPolicies in Enforce mode (disallow-latest-tag, require-resource-limits, disallow-privileged) (2026-07-14)
+- ✅ ArgoCD notifications + ApplicationSet git-directory generator deployed (2026-07-14)
+- ✅ Semaphore Ansible UI live at semaphore.apps.lab.home.arpa; 5 task templates (2026-07-18)
+- ✅ OpenTofu state backend: Minio `tofu-state` bucket; vms module codifies gitlab-1 (2026-07-18)
+- ✅ n150-1 KVM host fixes: bridge.conf, qemu-bridge-helper setuid, AppArmor NUMA rules captured in bootstrap-kvm.yml (2026-07-18)
+- ⚠ sudo password on n150-1/n150-2 exposed in terminal output (2026-07-18) — rotate via rotate-passwords.yml
+- Kyverno cleanup pending: Ollama image pin + Whisper versioned tag (required by disallow-latest-tag policy)
+- Authelia health stuck Progressing in ArgoCD — investigate(2