From 5f6d8ecf8dea7a8395d85e6fb79a2fc4653566ac Mon Sep 17 00:00:00 2001 From: Scott Wares Date: Sat, 18 Jul 2026 21:14:29 +0000 Subject: [PATCH 1/2] docs: update to 2026-07-18 lab state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HARDWARE.md: - Add gitlab-1 VM (n150-1, 192.168.1.50, UUID 6ea193a5, tofu/vms/) - Update last-verified date to 2026-07-18 - Fix N150 #3 WinRM item (was open, now closed) - Add Kyverno, Semaphore, ArgoCD ApplicationSet, OpenTofu milestones - Flag sudo password rotation needed - Note Kyverno cleanup + Authelia Progressing still pending services.md: - OpenTofu DNS: mark parked (Pi-hole v6 incompatible with provider) - OpenTofu VMs: mark ✓ with gitlab-1 details - NFS shared VM storage: ○ → ✓ UPDATES.md: - Fix Windows OS row: n150-1/2 are Ubuntu, only n150-3 is Windows --- docs/HARDWARE.md | 21 +++++++++++++++------ docs/UPDATES.md | 4 ++-- docs/services.md | 5 +++-- 3 files changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/HARDWARE.md b/docs/HARDWARE.md index 0cb6fdf..0a9292b 100644 --- a/docs/HARDWARE.md +++ b/docs/HARDWARE.md @@ -1,7 +1,7 @@ # Hardware Inventory & Roles Subnet: `192.168.1.0/24`. Gateway: `192.168.1.1`. -Last verified: 2026-07-10. +Last verified: 2026-07-18. > **Security:** Default passwords (`pi`/`odroid`/`root`) have been rotated on all managed > hosts via `ansible/playbooks/rotate-passwords.yml`. Credentials are in Ansible Vault — @@ -18,7 +18,7 @@ Last verified: 2026-07-10. | **N150 mini PC #2** | Intel N150 4C x86 | 16 GB | `192.168.1.21` (br0) | — | KVM hypervisor · Ubuntu 24.04 | | **N150 mini PC #3 (HTPC)** | Intel N150 4C x86 | 16 GB | — | `192.168.1.176` | Living-room HTPC | | **RPi 5** | Cortex-A76 4C | 8 GB | `192.168.1.128` | `192.168.1.124` (avoid) | HashiCorp Vault | -| **RPi 4B** | Cortex-A72 4C | 8 GB | `192.168.1.116` | — | Home Assistant host (Debian 12 Bookworm) | +| **RPi 4B** | Cortex-A72 4C | 8 GB | `192.168.1.116` | — | Pi-hole secondary DNS (Debian 12 Bookworm) | | **RPi 3B #2** | Cortex-A53 4C | 1 GB | `192.168.1.148` | `192.168.1.152` (avoid) | DNS primary (Pi-hole v6.4.3, Bookworm) | | **RPi 3B #1** | Cortex-A53 4C | 1 GB | — | — | ⚠ on-board power fault — retired | | **Odroid-XU3 #1** | Exynos5422 8C | 2 GB | `192.168.1.64` | — | Build agent (Python <3.8 — excluded from Ansible auto-updates) | @@ -33,6 +33,7 @@ Last verified: 2026-07-10. | VM | Host | IP | OS | Role | |----|------|----|----|------| +| **gitlab-1** | n150-1 | `192.168.1.50` | Ubuntu 22.04 | GitLab CE Omnibus · `gitlab.lab.home.arpa` · 80 GiB qcow2 at `/var/lib/libvirt/images/gitlab-1.qcow2` · MAC `52:54:00:6b:ab:01` · UUID `6ea193a5-61f9-4b65-8ee1-a90b343aec5f` · codified in `tofu/vms/` | | **ldap-1** | n150-1 | `192.168.1.70` | Ubuntu 24.04 | ~~lldap v0.6.3~~ **decommissioned 2026-07-04** — lldap migrated to k3s `lldap` namespace | ## Network @@ -130,10 +131,18 @@ CoreDNS extended with `coredns-custom` ConfigMap for in-cluster `*.apps.lab.home - ✅ n150-1/n150-2 sudo password: Ansible-Vault-encrypted in group_vars/kvm_hosts/secrets.yml; sync-secrets-to-vault.yml now backs it up to secret/lab/hosts (2026-07-03) - ✅ ArgoCD deploy key: sync-secrets-to-vault.yml backs up ~/.ssh/argocd-deploy-key to secret/lab/argocd (2026-07-03) - ✅ zswap enabled on n150-1/n150-2: zstd compressor, zsmalloc zpool, 20% max pool (2026-07-03) -- N150 #3: WinRM credentials (wrong password on file) -- ✅ Authelia → PostgreSQL: postgres.yaml added, configmap+deployment updated, replicas:2 — pending password generation + git push (2026-07-03) +- ✅ N150 #3: WinRM credentials corrected (2026-07-13) +- ✅ Authelia → PostgreSQL: postgres.yaml added, configmap+deployment updated (2026-07-03) - ✅ Shared NFS storage between n150-1/n150-2: /srv/libvirt-shared, libvirt-shared pool active on both nodes, SSH key exchange complete — VM live migration ready (2026-07-03) - ✅ Monitoring stack (Prometheus, Grafana, Alertmanager, Loki) migrated from odroid-nas to n150-1 (2026-07-04) - ✅ immich-library PV: hostPath+nodeAffinity → NFS ReadWriteMany; immich-server now schedules on any node (2026-07-04) -- ✅ lldap migrated from ldap-1 KVM VM (192.168.1.70) to k3s Deployment in lldap namespace; ldap-1 VM stopped (2026-07-04) -- ✅ MQTT HA: opi-zero2w-4 (.99) deployed as secondary broker bridging all topics to/from opi-zero2w-2 (.188); M5Stack firmware has automatic failover (2 \ No newline at end of file +- ✅ lldap migrated from ldap-1 KVM VM (192.168.1.70) to k3s Deployment in lldap namespace; ldap-1 VM decommissioned (2026-07-04) +- ✅ MQTT HA: opi-zero2w-4 (.99) deployed as secondary broker bridging all topics to/from opi-zero2w-2 (.188); M5Stack firmware has automatic failover (2026-07-10) +- ✅ Kyverno: 3 ClusterPolicies in Enforce mode (disallow-latest-tag, require-resource-limits, disallow-privileged) (2026-07-14) +- ✅ ArgoCD notifications + ApplicationSet git-directory generator deployed (2026-07-14) +- ✅ Semaphore Ansible UI live at semaphore.apps.lab.home.arpa; 5 task templates (2026-07-18) +- ✅ OpenTofu state backend: Minio `tofu-state` bucket; vms module codifies gitlab-1 (2026-07-18) +- ✅ n150-1 KVM host fixes: bridge.conf, qemu-bridge-helper setuid, AppArmor NUMA rules captured in bootstrap-kvm.yml (2026-07-18) +- ⚠ sudo password on n150-1/n150-2 exposed in terminal output (2026-07-18) — rotate via rotate-passwords.yml +- Kyverno cleanup pending: Ollama image pin + Whisper versioned tag (required by disallow-latest-tag policy) +- Authelia health stuck Progressing in ArgoCD — investigate(2 \ No newline at end of file diff --git a/docs/UPDATES.md b/docs/UPDATES.md index 40efe8c..0f9dd63 100644 --- a/docs/UPDATES.md +++ b/docs/UPDATES.md @@ -15,7 +15,7 @@ Last updated: 2026-07-18. | Pi-hole application | Ansible `update-non-apt.yml -t pihole` in GitHub Actions | Weekly (Sunday 03:00 Central)¹ | No | | HashiCorp Vault binary | apt (HashiCorp repo) — included in OS play; seal-check play runs after | Weekly¹ | Yes — unseal manually if sealed after restart | | **KVM VMs (gitlab-1, etc.)** | **`sandbox-vm-update.yml` — clone-test-promote pipeline** | **Manual / on-demand** | **Yes — human runs promote step** | -| Windows OS (n150-1/2/3) | None — Windows Update runs uncontrolled | Uncontrolled | Manual if policy required | +| Windows OS (n150-3 / yikw) | None — Windows Update runs uncontrolled | Uncontrolled | Manual if policy required | ¹ GitHub Actions schedule runs on the **self-hosted runner on H4** (`runs-on: [self-hosted, lab]`) which has direct LAN access. Register the runner: @@ -219,7 +219,7 @@ to confirm before assuming External Secrets and dependent workloads are healthy. --- -## 6. Windows OS (n150-1, n150-2, n150-3) +## 6. Windows OS (n150-3 / yikw only — n150-1/2 are Ubuntu) **Not currently automated.** Windows Update runs on its own schedule (or not, depending on local policy). The Ansible `windows-bootstrap.yml` playbook handles initial setup but diff --git a/docs/services.md b/docs/services.md index 5f52468..e1aa0fc 100644 --- a/docs/services.md +++ b/docs/services.md @@ -34,7 +34,7 @@ Everything the design implies, so you can tick what's needed and spot gaps. Stat | Service | Status | Placement | Notes | |---------|:------:|-----------|-------| | KVM / libvirt | ✓ | n150-1 (.42) + n150-2 (.21) | bare-metal Ubuntu 24.04 hypervisors; cloud-init VMs | -| NFS storage for VMs | ○ | H4 exports → n150 hosts | shared VM disk storage | +| NFS storage for VMs | ✓ | H4 exports `/srv/libvirt-shared` → n150-1 + n150-2 | `libvirt-shared` pool active on both nodes; enables VM live migration | | n150-3 (yikw) | ✓ | Windows HTPC | TV/browse; not a hypervisor | ## Platform / CaaS (L2) @@ -79,7 +79,8 @@ Everything the design implies, so you can tick what's needed and spot gaps. Stat | Service | Status | Placement | Notes | |---------|:------:|-----------|-------| | Ansible | ✓ | control node | host config; `ansible/playbooks/`; vault-encrypted secrets | -| OpenTofu (DNS pilot) | ● | `tofu/dns/` | Pi-hole DNS records via `ryanwholey/pihole ~>0.2`; state in Minio `tofu-state` bucket; node IPs need updating before first apply | +| OpenTofu — VMs | ✓ | `tofu/vms/` | gitlab-1 VM codified (UUID `6ea193a5`); `ignore_changes = all` + `prevent_destroy`; state in Minio `tofu-state/vms/terraform.tfstate`; init with `~/.tofu-backend.hcl` | +| OpenTofu — DNS | ● | `tofu/dns/` | Parked — `ryanwholey/pihole v0.2` uses Pi-hole v5 session API; incompatible with Pi-hole v6 on octopi. DNS managed via Ansible `playbooks/dns.yml` instead. | | KVM + cloud-init | ✓ | n150-1/2 | VMs via `virt-install`; SSH-ready | | rpi-imager / cloud-init | ● | flashing | SD-card SBCs | | PlatformIO / ESPHome / OTA | ● | firmware | microcontrollers | From 43f8e573c63a6fa394d42f0d0a948da1203c6740 Mon Sep 17 00:00:00 2001 From: Scott Wares Date: Sat, 18 Jul 2026 23:32:30 +0000 Subject: [PATCH 2/2] update --- CLAUDE.md | 8 ++-- README.md | 108 +++++++++++++++++++++++++++++-------------- docs/ARCHITECTURE.md | 24 ++++++++-- docs/OVERVIEW.md | 48 +++++++++++-------- docs/RUNBOOK.md | 8 ++-- docs/UPDATES.md | 2 +- 6 files changed, 134 insertions(+), 64 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b9a3834..de8267b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,8 @@ # CLAUDE.md — operating rules for this lab -You are helping operate a home lab whose core is an **Odroid-H4 Ultra** that is *both* a -NAS and a single-node **k3s** cluster (Ubuntu 22.04). Read this before acting. Full context -is in `docs/` (start with `ARCHITECTURE.md`). +You are helping operate a home lab whose core is an **Odroid-H4 Ultra** NAS, part of a +**3-node HA k3s cluster** (H4 + n150-1 + n150-2 as servers; opi5pro-1/2 as ARM64 agents). +Read this before acting. Full context is in `docs/` (start with `ARCHITECTURE.md`). ## How changes are made @@ -65,7 +65,7 @@ is in `docs/` (start with `ARCHITECTURE.md`). The H4 is the core (k3s server + NAS, Ubuntu 22.04, `192.168.1.160`). The two **Orange Pi 5 Pro** boards (8C/16GB/NPU) are k3s agents / AI inference hosts; RPi 5 -runs Vault; RPi 4B runs Pi-hole (secondary DNS, 192.168.1.116); lldap runs as a k3s Deployment in the `lldap` namespace (ldap-1 VM decommissioned +runs Vault; RPi 4B runs Pi-hole (secondary DNS, 192.168.1.116); Home Assistant runs as a k3s Deployment in the `home-assistant` namespace; lldap runs as a k3s Deployment in the `lldap` namespace (ldap-1 VM decommissioned 2026-07-04); the XU3 is a build agent. DNS needs a permanent host. M5Stack + OPi NPUs are edge inference endpoints, not cluster nodes. The map's plaintext credentials must be rotated. See `docs/HARDWARE.md`. \ No newline at end of file diff --git a/README.md b/README.md index 601d96f..2e19521 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # Home Lab — DevOps Environment -A GitOps-managed home lab built around an **Odroid-H4 Ultra** that serves double duty as -a NAS and a single-node **k3s** cluster, with an ARM inference fleet. Infrastructure is -defined as code: **Ansible** stands hosts up, **Argo CD** runs everything inside the -cluster from this git repo, and a two-tier storage model (hot NVMe + two cold SATA RAID 1 -mirrors) keeps data safe. +A GitOps-managed home lab built around an **Odroid-H4 Ultra** NAS core and a +**3-node HA k3s cluster** (H4 + two N150 mini PCs), with an ARM64 inference fleet +(two Orange Pi 5 Pros). Infrastructure is defined as code: **Ansible** stands hosts up, +**Argo CD** runs everything inside the cluster from this git repo, and a two-tier storage +model (hot NVMe + two cold SATA RAID 1 mirrors) keeps data safe. ## Why it's shaped this way @@ -34,47 +34,71 @@ mirrors) keeps data safe. | App | Namespace | Notes | |-----|-----------|-------| -| Immich | `immich` | Photo server + Postgres + Redis + ML; library on cold RAID at `/mnt/cold-8t/immich` | -| LiteLLM gateway | `ai-gateway` | Load-balances `chat` across both Ollama instances (simple-shuffle) | -| Ollama | `ai-gateway` | qwen2.5:3b on opi5pro-1, pinned via nodeSelector | -| Ollama-2 | `ai-gateway` | qwen2.5:3b on opi5pro-2, pinned via nodeSelector | -| kube-prometheus-stack | `monitoring` | Prometheus (30d/50GB), Grafana, Alertmanager, node-exporter on all nodes | +| Immich | `immich` | Photo server + Postgres (vectorchord) + Redis + ML; library on NFS ReadWriteMany PV | +| LiteLLM gateway | `ai-gateway` | Unified OpenAI-compatible API (`ai.apps.lab.home.arpa`) across all backends | +| RKLLama | `ai-gateway` | NPU-native LLM on opi5pro-1/2 (DeepSeek-R1-Distill-Qwen-1.5B, ~7–8 tok/s) | +| Ollama | `ai-gateway` | In-cluster fallback engine on opi5pro-1/2; pinned to `ollama/ollama:0.32.0` | +| m5stack-adapter | `ai-gateway` | OpenAI shim for M5Stack `/api/*` protocol; image 0.1.1 | +| Whisper STT | `whisper` | Speech-to-text at `https://stt.apps.lab.home.arpa`; CPU on n150-1 | +| lldap | `lldap` | Lightweight LDAP directory; web UI at `lldap.apps.lab.home.arpa` | +| Authelia | `authelia` | OIDC/SSO backed by lldap; PostgreSQL backend; `authelia.apps.lab.home.arpa` | +| Home Assistant | `home-assistant` | `ha.apps.lab.home.arpa`; MQTT consumer (broker at opi-zero2w-2 .188) | +| Minio | `minio` | S3-compatible object store; `tofu-state` bucket holds OpenTofu state | +| Semaphore | `semaphore` | Ansible UI at `semaphore.apps.lab.home.arpa` | +| Kyverno | `kyverno` | 3 ClusterPolicies in Enforce mode (no-latest-tag, resource-limits, no-privileged) | +| kube-prometheus-stack | `monitoring` | Prometheus (30d/40GB), Grafana, Alertmanager, Loki, Alloy on all nodes | | external-secrets | `external-secrets` | Pulls secrets from Vault (KV v2 at `secret/lab/`) | -| cert-manager | `cert-manager` | TLS certificate management | -| Argo CD | `argocd` | GitOps controller — manages all of the above | +| cert-manager | `cert-manager` | TLS via `lab-ca` ClusterIssuer (self-signed root CA) | +| Argo CD | `argocd` | GitOps controller — selfHeal + prune on all apps | ### Host services (outside k3s) | Service | Host | IP | |---------|------|----| -| HashiCorp Vault v1.21.0 | rpi5 | 192.168.1.128 | -| Mosquitto MQTT | opi-zero2w-2 | 192.168.1.188 | -| Pi-hole DNS (primary) | octopi | 192.168.1.148 | +| HashiCorp Vault | rpi5 | 192.168.1.128 | +| Mosquitto MQTT (primary) | opi-zero2w-2 | 192.168.1.188 | +| Mosquitto MQTT (secondary, HA bridge) | opi-zero2w-4 | 192.168.1.99 | +| Pi-hole DNS (primary, v6.4.3) | octopi (RPi 3B #2) | 192.168.1.148 | +| Pi-hole DNS (secondary, v6) | rpi4b (RPi 4B) | 192.168.1.116 | +| dnsmasq DNS (tertiary fallback) | opi-zero2w-1 | 192.168.1.184 | | Samba / NFS (NAS) | H4 (host) | 192.168.1.160 | +| GitLab CE | gitlab-1 VM on n150-1 | 192.168.1.50 | ### Ingress endpoints | URL | Service | |-----|---------| -| `grafana.apps.lab.home.arpa` | Grafana | | `argocd.apps.lab.home.arpa` | Argo CD | +| `grafana.apps.lab.home.arpa` | Grafana | | `immich.apps.lab.home.arpa` | Immich | -| `*.apps.lab.home.arpa` | → 192.168.1.160 (Traefik) | +| `ai.apps.lab.home.arpa` | LiteLLM gateway | +| `ha.apps.lab.home.arpa` | Home Assistant | +| `authelia.apps.lab.home.arpa` | Authelia (SSO) | +| `lldap.apps.lab.home.arpa` | lldap (LDAP directory UI) | +| `semaphore.apps.lab.home.arpa` | Semaphore (Ansible UI) | +| `minio.apps.lab.home.arpa` | Minio (S3 API) | +| `minio-console.apps.lab.home.arpa` | Minio console | +| `stt.apps.lab.home.arpa` | Whisper STT | +| `*.apps.lab.home.arpa` | → 192.168.1.160 (Traefik ingress) | ## Fleet | Host | Ansible name | IP | Role | |------|--------------|----|------| -| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS | -| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, Ollama inference | -| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, Ollama-2 inference | -| Raspberry Pi 5 | rpi5 | 192.168.1.128 | Vault | -| N150 mini PC #1 | n150-1 | 192.168.1.42 | Ubuntu 24.04, k3s server + KVM hypervisor | -| N150 mini PC #2 | n150-2 | 192.168.1.21 | Ubuntu 24.04, k3s server + KVM hypervisor | -| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed, sleep disabled) | -| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT broker | -| OctoPi (RPi 3B #2) | octopi | 192.168.1.148 | Pi-hole DNS (Bookworm, Pi-hole v6.4.3) | -| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent (flagged unstable) | +| Odroid-H4 Ultra | h4-core | 192.168.1.160 | k3s server + NAS (smbd/NFS) | +| N150 mini PC #1 | n150-1 | 192.168.1.42 | k3s server + KVM hypervisor (Ubuntu 24.04) | +| N150 mini PC #2 | n150-2 | 192.168.1.21 | k3s server + KVM hypervisor (Ubuntu 24.04) | +| Orange Pi 5 Pro #1 | opi5pro-1 | 192.168.1.168 | k3s agent, RKLLama/Ollama inference, NPU | +| Orange Pi 5 Pro #2 | opi5pro-2 | 192.168.1.172 | k3s agent, RKLLama/Ollama inference, NPU | +| Raspberry Pi 5 | rpi5 | 192.168.1.128 | HashiCorp Vault | +| Raspberry Pi 4B | rpi4b | 192.168.1.116 | Pi-hole secondary DNS (v6, Bookworm) | +| RPi 3B #2 (octopi) | octopi | 192.168.1.148 | Pi-hole primary DNS (v6.4.3, Bookworm) | +| N150 mini PC #3 | n150-3 | 192.168.1.176 | Windows HTPC (WinRM managed) | +| OPi Zero 2W #1 | opi-zero2w-1 | 192.168.1.184 | dnsmasq DNS tertiary fallback | +| OPi Zero 2W #2 | opi-zero2w-2 | 192.168.1.188 | MQTT primary broker | +| OPi Zero 2W #3 | opi-zero2w-3 | 192.168.1.217 | dnsmasq DNS secondary (Armbian Trixie) | +| OPi Zero 2W #4 | opi-zero2w-4 | 192.168.1.99 | MQTT secondary broker (HA bridge) | +| Odroid XU3 | xu3-1 | 192.168.1.64 | Build agent | ## Quickstart (fresh bootstrap) @@ -134,13 +158,29 @@ See [docs/SECURITY.md](docs/SECURITY.md) for the full security model. ## TODO ### Actionable now -- [ ] Rotate octopi Pi-hole credentials (currently plaintext) -- [ ] Flash octopi to Raspberry Pi OS Bookworm → upgrade to Pi-hole v6 -- [ ] Move photo library (~1.3TB) into `/mnt/cold-8t/immich` so Immich can serve it +- [ ] Rotate sudo passwords on n150-1/n150-2 (exposed in terminal output 2026-07-18 — run `rotate-passwords.yml`) +- [ ] Upgrade rknpu driver 0.9.6 → 0.9.7 to unlock 3B+ model support on opi5pro-1/2 +- [ ] Fix Ollama image pin + Whisper versioned tag (required by Kyverno `disallow-latest-tag` policy) +- [ ] Investigate Authelia health stuck Progressing in ArgoCD ### Longer term - [ ] Vault TLS — currently plain HTTP; wire cert-manager before exposing beyond LAN -- [ ] LDAP integration with Immich SSO once OpenLDAP is up -- [ ] n150-1/2 → OVMS Intel iGPU for embeddings (`embeddings` model in LiteLLM config already wired) -- [ ] Offsite restic backup target -- [ ] Pin Immich image tag (currently `:release` floating) — Renovate will open the first PR +- [ ] Windows Update automation (`windows-updates.yml`) — `ansible.windows.win_updates` module ready, playbook not yet written +- [ ] Remote access — WireGuard or Tailscale + +### Done ✅ +- [x] octopi flashed to Bookworm, Pi-hole v6.4.3 running (2026-07-13) +- [x] RPi 4B: Pi-hole v6 secondary DNS live at 192.168.1.116 (2026-07-02) +- [x] n150-1/n150-2 joined as k3s server nodes, kube-vip VIP 192.168.1.200 (2026-07-02) +- [x] lldap migrated from ldap-1 VM to k3s Deployment in `lldap` namespace (2026-07-04) +- [x] Authelia → PostgreSQL backend (2026-07-03) +- [x] Immich library → NFS ReadWriteMany PV (schedules on any node) (2026-07-04) +- [x] Shared NFS storage between n150-1/n150-2 for VM live migration (2026-07-03) +- [x] Monitoring stack migrated to n150-1 (2026-07-04) +- [x] zswap on n150-1/n150-2 (zstd, zsmalloc, 20%) (2026-07-03) +- [x] MQTT HA: opi-zero2w-4 secondary broker with bidirectional bridge (2026-07-10) +- [x] Kyverno 3 ClusterPolicies in Enforce mode (2026-07-14) +- [x] ArgoCD notifications + git-directory ApplicationSet (2026-07-14) +- [x] Semaphore Ansible UI live (2026-07-18) +- [x] OpenTofu state in Minio; gitlab-1 VM codified (2026-07-18) +- [x] Offsite restic backup target (backup-offsite.timer) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1c8764a..7cf8262 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -133,13 +133,14 @@ The lab is a flat **192.168.1.0/24** network. **DNS is the linchpin of the insta k3s needs `api.lab.home.arpa → 192.168.1.200` (kube-vip VIP) and `*.apps.lab.home.arpa → 192.168.1.160` (H4 Traefik ingress). -Three DNS servers provide redundancy: +Four DNS servers provide redundancy: | Role | Host | IP | Engine | |------|------|----|--------| -| Primary | RPi 3B #2 (octopi) | `192.168.1.148` | Pi-hole (Bookworm pending) | -| Secondary | RPi 4B | `192.168.1.116` | Pi-hole v6 | +| Primary | RPi 3B #2 (octopi) | `192.168.1.148` | Pi-hole v6.4.3 (Bookworm, confirmed 2026-07-13) | +| Secondary | RPi 4B | `192.168.1.116` | Pi-hole v6 (Bookworm) | | Tertiary | OPi Zero 2W #1 | `192.168.1.184` | dnsmasq (fallback) | +| Quaternary | OPi Zero 2W #3 | `192.168.1.217` | dnsmasq (Armbian Trixie, 2026-07-10) | Inside the cluster, CoreDNS has a custom zone (`coredns-custom` ConfigMap in kube-system) that answers all `*.apps.lab.home.arpa` queries with `192.168.1.160`, ensuring pods on any @@ -198,3 +199,20 @@ Grouped services and the hosts that provide them: An **interactive, filterable** version (toggle hardware classes, select-all/clear) is at [service-map.html](service-map.html) — open it in a browser. A **host-centric** companion (one card per box, everything it runs, same filters) is at [host-map.html](host-map.html). + +### Key cluster services (as of 2026-07-18) + +| Service | Namespace | Notes | +|---------|-----------|-------| +| Argo CD | `argocd` | GitOps; app-of-apps + git-directory ApplicationSet | +| Immich | `immich` | Photos/video; NFS ReadWriteMany library PV | +| Home Assistant | `home-assistant` | HA automation; MQTT consumer | +| lldap | `lldap` | Lightweight LDAP; migrated from ldap-1 VM (2026-07-04) | +| Authelia | `authelia` | OIDC/SSO; PostgreSQL backend | +| LiteLLM + AI backends | `ai-gateway` | RKLLama (NPU), Ollama (fallback), m5stack-adapter, Whisper | +| Semaphore | `semaphore` | Ansible UI; runs playbooks from inside the cluster | +| Minio | `minio` | S3 object store; holds OpenTofu state | +| Kyverno | `kyverno` | Policy enforcement (3 ClusterPolicies, Enforce mode) | +| kube-prometheus-stack | `monitoring` | Prometheus + Grafana + Alertmanager + Loki + Alloy | +| External Secrets | `external-secrets` | Syncs Vault KV v2 → k8s Secrets | +| cert-manager | `cert-manager` | `lab-ca` ClusterIssuer; signs `*.apps.lab.home.arpa` TLS | diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md index 3166e3b..b8caed7 100644 --- a/docs/OVERVIEW.md +++ b/docs/OVERVIEW.md @@ -1,14 +1,17 @@ # Lab Overview -Current as of 2026-07-07. Detailed service catalog in [SERVICES.md](SERVICES.md), hardware in [HARDWARE.md](HARDWARE.md). +Current as of 2026-07-18. Detailed service catalog in [services.md](services.md), hardware in [HARDWARE.md](HARDWARE.md). ## Infrastructure & Platform -- **k3s cluster** — H4, n150-1, n150-2 as servers; opi5pro-1 and opi5pro-2 as agents; `local-path` StorageClass on NVMe -- **ArgoCD** — GitOps from `gitops/`; selfHeal + prune on all apps +- **k3s cluster** — 3-node HA control plane: H4, n150-1, n150-2 as servers; opi5pro-1 and opi5pro-2 as ARM64 agents; `local-path` StorageClass on NVMe +- **ArgoCD** — GitOps from `gitops/`; selfHeal + prune on all apps; git-directory ApplicationSet for workloads - **Traefik** — ingress for all `*.apps.lab.home.arpa`; TLS via cert-manager lab CA - **kube-vip** — control-plane HA VIP (`192.168.1.200`) -- **KVM/libvirt** — bare-metal hypervisors on n150-1 and n150-2 +- **KVM/libvirt** — bare-metal hypervisors on n150-1 and n150-2; shared NFS storage at `/srv/libvirt-shared` for VM live migration +- **OpenTofu** — VMs module codifies gitlab-1; state in Minio `tofu-state` bucket +- **Semaphore** — Ansible UI at `semaphore.apps.lab.home.arpa`; runs playbooks from a k3s pod +- **Kyverno** — 3 ClusterPolicies in Enforce mode: disallow-latest-tag, require-resource-limits, disallow-privileged-containers ## Identity & Security @@ -19,7 +22,7 @@ Current as of 2026-07-07. Detailed service catalog in [SERVICES.md](SERVICES.md) ## Networking & DNS -- **Pi-hole** — primary DNS + ad-blocking on octopi; dnsmasq fallback on opi-zero2w-1 +- **Pi-hole** — primary DNS + ad-blocking on octopi (192.168.1.148); secondary on RPi 4B (192.168.1.116); dnsmasq fallback on opi-zero2w-1 - **CoreDNS custom zone** — wildcard `*.apps.lab.home.arpa → 192.168.1.160` for in-cluster resolution - **Mosquitto MQTT** — opi-zero2w-2 primary, opi-zero2w-4 HA bridge @@ -34,16 +37,20 @@ Current as of 2026-07-07. Detailed service catalog in [SERVICES.md](SERVICES.md) ## AI & Inference - **LiteLLM gateway** — unified OpenAI-compatible API across all backends (`ai.apps.lab.home.arpa`) -- **RKLLama** — NPU-native LLM on opi5pro-1 and opi5pro-2 (~7–8 tok/s, RK3588S) -- **OpenVINO Model Server** — currently disabled (pending GPU runtime + model IR setup); planned for H4 + n150s Intel iGPU -- **Ollama** — in-cluster CPU fallback (opi5pro-1 and opi5pro-2) -- **M5Stack escalation router** — 3-tier edge AI: local NPU → Claude API → Claude Code -- **m5stack-adapter** — OpenAI-compatible shim for M5Stack `/api/*` protocol +- **RKLLama** — NPU-native LLM on opi5pro-1 and opi5pro-2; running DeepSeek-R1-Distill-Qwen-1.5B_w8a8_g128 (~7–8 tok/s); rknpu driver 0.9.6 caps NPU alloc to ~2.2 GB (upgrade to 0.9.7 to unlock 3B+ models) +- **Ollama** — in-cluster fallback engine on opi5pro-1/2; pinned to `ollama/ollama:0.32.0` +- **Whisper STT** — speech-to-text at `https://stt.apps.lab.home.arpa/v1/audio/transcriptions`; CPU on n150-1; base model +- **OpenVINO Model Server** — currently disabled; crash-looping on n150-1, scaled to 0; planned for H4 + n150s Intel iGPU +- **M5Stack escalation router** — 3-tier edge AI: local NPU → Claude API → Claude Code orchestrator +- **m5stack-adapter** — OpenAI-compatible shim for M5Stack `/api/*` protocol (image 0.1.1) +- **Claude Code orchestrator** — live on opi5pro-1, HTTPS :8443; Tier 3 escalation ## Applications -- **Immich** — self-hosted photo/video library; SSO via Authelia OIDC; smart search + face recognition; Postgres with vector extension (`immich.apps.lab.home.arpa`) -- **Private container registry** — in-cluster image hosting +- **Immich** — self-hosted photo/video library; SSO via Authelia OIDC; smart search + face recognition; Postgres with vectorchord extension; library on NFS ReadWriteMany PV (`immich.apps.lab.home.arpa`) +- **Home Assistant** — k3s Deployment in `home-assistant` namespace (`ha.apps.lab.home.arpa`); MQTT consumer (broker at opi-zero2w-2 .188) +- **Minio** — S3-compatible object store; `tofu-state` bucket for OpenTofu state; `minio.apps.lab.home.arpa` +- **GitLab CE** — gitlab-1 VM on n150-1; `gitlab.lab.home.arpa`; runner token in Vault ## Storage & Backup @@ -53,13 +60,16 @@ Current as of 2026-07-07. Detailed service catalog in [SERVICES.md](SERVICES.md) ## Provisioning & CI -- **Ansible** — all host config; playbooks for bootstrap, updates, k3s, DNS, zswap, etc. -- **GitHub Actions** — self-hosted runner on H4; weekly scheduled maintenance (OS updates, Pi-hole, Vault seal check, zswap) +- **Ansible** — all host config; playbooks for bootstrap, updates, k3s, DNS, zswap, KVM bootstrap, sandbox-vm-update, password rotation, secrets sync +- **Semaphore** — Ansible UI; runs playbooks against the lab from inside the cluster +- **GitHub Actions** — self-hosted runner on H4; weekly scheduled maintenance (OS updates, Pi-hole, Vault seal check, lab health check) +- **OpenTofu** — VMs module (`tofu/vms/`) codifies gitlab-1; DNS module parked (Pi-hole v5 provider incompatible with v6) ## Deferred / Gaps -- Remote access — WireGuard or Tailscale -- UPS + NUT (H4 is a single-point storage risk) -- Homepage / dashboard -- CloudNativePG + Redis operators for shared DB -- RPi 4B runs Pi-hole secondary DNS (192.168.1.116) +- Remote access — WireGuard or Tailscale (very low priority) +- UPS + NUT — H4 is a single-point storage risk; cheap insurance +- Vault TLS — currently plain HTTP; add cert-manager cert before exposing beyond LAN +- Windows Update automation — `windows-updates.yml` playbook not yet written +- rknpu driver 0.9.7 upgrade — unlocks 3B+ model support on opi5pro NPUs +- OpenVINO OVMS — crash-looping on n150-1, scaled to 0; blocked on GPU runtime fix diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index fbbcf1f..3323ba8 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -96,8 +96,10 @@ refresh per secret with: kubectl get applications -n argocd # all Synced/Healthy systemctl status backup-nas.timer backup-etcd.timer ssh swares@192.168.1.128 systemctl status backup-vault.timer # rpi5 - # backup-lldap.timer — lldap is now a k3s Deployment; backup is via Immich DB CronJob pattern - # (ldap-1 VM decommissioned 2026-07-04; timer no longer applies) + # Note: lldap is now a k3s Deployment (ldap-1 VM decommissioned 2026-07-04). + # The backup-lldap.timer on ldap-1 no longer exists. Back up lldap's SQLite PVC + # via a k8s CronJob if needed — the PV is on local-path NVMe on whichever node + # the lldap pod lands on. --- @@ -119,7 +121,7 @@ Never `kubectl apply` directly against main — it drifts and ArgoCD reverts it. | `backup-nas` | daily 01:30 | restic of `/srv/nas` + `/mnt/cold-8t/VMs` + `/mnt/cold-8t/immich` → cold-8t, then `restic copy` → cold-sec + offsite | none | | `backup-etcd` | daily | k3s SQLite state → `/mnt/cold-8t/k3s-etcd-snapshots/`, 7 copies retained | none | | `backup-vault` | daily 02:30 | Vault raft snapshot → `/mnt/cold-8t/vault-snapshots/`, 30-day retention | none | -| `backup-lldap` | ~~daily 02:45~~ | ~~lldap SQLite on ldap-1 VM~~ — **ldap-1 decommissioned**; lldap now in k3s; add a k8s CronJob to back up the PVC SQLite | — | +| `backup-lldap` | — | **ldap-1 VM decommissioned 2026-07-04.** lldap now runs as a k3s Deployment in the `lldap` namespace; SQLite data is on a `local-path` PVC. To back it up, add a k8s CronJob that copies the SQLite file from the PVC mount. | — | | Immich DB dump | daily 01:30 | `pg_dump` via k8s CronJob → `/mnt/cold-8t/immich/backups/` (captured by restic above) | none | Check: diff --git a/docs/UPDATES.md b/docs/UPDATES.md index 0f9dd63..1fde16a 100644 --- a/docs/UPDATES.md +++ b/docs/UPDATES.md @@ -331,4 +331,4 @@ Requires per-VM inventory vars (`kvm_host`, `libvirt_vm_name`, `health_check_url | Vault TLS | Medium | Currently plain HTTP; add before exposing beyond LAN | | ~~octopi OS upgrade (Raspbian Buster → Bookworm)~~ | ✅ Done (2026-07-13) | Bookworm + Pi-hole v6.4.3/FTL v6.7 confirmed running | | ~~zswap on n150-1/n150-2~~ | ✅ Done (2026-07-03) | zswap enabled: zstd compressor, zsmalloc zpool, 20% max pool | -| Shared storage (n150-1 ↔ n150-2) | Low | NFS or Ceph enables VM live migration between hypervisors | +| ~~Shared storage (n150-1 ↔ n150-2)~~ | ✅ Done (2026-07-03) | NFS `/srv/libvirt-shared` exported from H4; libvirt-shared pool active on both nodes; VM live migration ready |