feat(healthcheck): decouple host liveness from the self-update loop

CHANGELOG.md

@@ -18,6 +18,7 @@ module follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Every subkey ships a module default and the groups merge deep across Hiera layers, so node data holds only its deviations, and a mistyped subkey now fails the compile instead of being silently ignored.
 - The runner-token store reads as part of the runner family: `tokens` is now `runner_tokens`, alongside `runners` and `runner_defaults`. The automatic `Sensitive` wrap on lookup moves with the new key; the host's secret store file must follow the rename.
 - The apply script now installs on any declared-standalone host (`standalone.manage: true`), not only with the self-update loop, so manual applies and the loop share the same single apply command; enabling the loop on a host not declared standalone fails at compile time.
+- The liveness healthcheck now installs on any declared-standalone host (`standalone.manage: true`), not only with the self-update loop, so a standalone host without the loop still has its manager service and rootless Docker daemon monitored (cadence `standalone.healthcheck_interval`); the loop's own supervision checks (apply-timer state, checkout staleness, bootstrap-gem presence) layer on only when the loop is enabled.
 - Freshly provisioned hosts now get a 165536-wide subordinate-ID range, which satisfies the ID-mapping requirement of nested rootless BuildKit builds.
 - Subordinate UID/GID ranges are now provisioned by `rootless_docker.manage`, together with the other rootless-Docker prerequisites, instead of by the runner-account toggle: rootless Docker can now be brought up on a host whose runner account is owned by another system. An existing range entry is never overwritten, and `runner_account.manage` keeps owning the group, user, and home.
 - The managed runner service's `DOCKER_HOST` is now module-owned and derived from `runner_account.uid`; `runner_service.environment` carries additional variables only and rejects a `DOCKER_HOST` line, and a managed service requires `runner_account.uid`.

README.md

@@ -233,8 +233,8 @@ ownership toggle, opt-in by default:
 | Runner account (group, user, home) | [`runner_account.manage`](#runner_account) | `false` |
 | Rootless-Docker daemon bring-up, subordinate IDs | [`rootless_docker.manage`](#rootless_docker) | `false` |
 | Runner service + its systemd drop-in | [`runner_service.manage`](#runner_service) | `false` |
-| Standalone topology (the apply script) | [`standalone.manage`](#standalone) | `false` |
-| Self-update loop + healthcheck | [`standalone.self_update.manage`](#standaloneself_update) | `false` |
+| Standalone topology (apply script + liveness healthcheck) | [`standalone.manage`](#standalone) | `false` |
+| Self-update loop | [`standalone.self_update.manage`](#standaloneself_update) | `false` |
 
 The `manage` keys are **persistent ownership switches**, not one-shot bootstrap flags:
 set once in the host's Hiera and left on, so every apply keeps owning and drift-correcting that
@@ -347,7 +347,10 @@ window ([Restarts and graceful shutdown](#restarts-and-graceful-shutdown)).
 rather than being deployed by a Puppet server. It installs the apply script
 (`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition of the apply command
 for manual runs and the self-update loop alike
-([Applying the configuration](#applying-the-configuration)). The apply's manifest, module
+([Applying the configuration](#applying-the-configuration)), and a liveness healthcheck
+(script + timer, default every `15min`, `standalone.healthcheck_interval`) that verifies the
+manager service and the rootless daemon (`docker info` as the runner user) — installed on any
+standalone host, with or without the loop. The apply's manifest, module
 directory and Hiera configuration derive from the documented control-repository layout
 (`puppet/manifests/site.pp`, `puppet/modules`, `puppet/hiera.yaml` beneath the checkout); the
 isolated Puppet state directories are `standalone.puppet_confdir`/`standalone.puppet_vardir`,
@@ -357,18 +360,18 @@ non-login apply.
 #### `standalone.self_update`
 
 With `standalone.self_update.manage` on — only valid on a standalone host: enabling it with
-`standalone.manage` off fails at compile time — the module installs the self-update loop, two
-units each on its own timer:
-
-- A oneshot systemd service + timer (default every `5min`, `standalone.self_update.apply_interval`)
-  that fetch the control-repository checkout, run `git verify-commit` on the remote branch
-  (only signed commits are applied), reset to it, install Puppetfile-pinned modules via
-  `r10k puppetfile install` (a no-op without a Puppetfile), and re-apply through the apply
-  script above.
-- A healthcheck script + timer (default every `15min`, `standalone.healthcheck_interval`) that
-  verifies the manager service, the rootless daemon (`docker info` as the runner user, from a
-  non-login context), and that the checkout is not stale against the remote (a dead pull
-  credential fails loud instead of leaving the host applying old code behind a green timer).
+`standalone.manage` off fails at compile time — the module installs the self-update loop: a
+oneshot systemd service + timer (default every `5min`, `standalone.self_update.apply_interval`)
+that fetch the control-repository checkout, run `git verify-commit` on the remote branch (only
+signed commits are applied), reset to it, install Puppetfile-pinned modules via
+`r10k puppetfile install` (a no-op without a Puppetfile), and re-apply through the apply script
+above.
+
+The loop also layers its own supervision onto the liveness healthcheck that `standalone.manage`
+installs: it adds checks that the apply timer is enabled and armed, that the checkout is not
+stale against the remote (a dead pull credential fails loud instead of leaving the host applying
+old code behind a green timer), and that the bootstrap gems (`r10k`, `hiera-eyaml`) are present
+in the AIO Ruby.
 
 The service sets `HOME=/root` (git/SSH need it) and an explicit `TimeoutStartSec`
 (`standalone.self_update.apply_timeout`); Puppet exit code 2 ("changes applied") counts as
@@ -386,7 +389,7 @@ host-specific:
 
 | Hiera key | Description | Default |
 |---|---|---|
-| `runner_account.uid` | Numeric uid of the runner account; the rootless runtime paths (`/run/user/<uid>`, the docker socket) derive from it | none — required when `runner_account.manage`, `rootless_docker.manage` or `standalone.self_update.manage` is on, or for a `socket_mount` runner |
+| `runner_account.uid` | Numeric uid of the runner account; the rootless runtime paths (`/run/user/<uid>`, the docker socket) derive from it | none — required when `runner_account.manage`, `rootless_docker.manage` or `standalone.manage` is on, or for a `socket_mount` runner |
 | `standalone.control_repository_path` | Checkout of the control repository on the host (the apply and self-update target) | `/opt/gitlab-runner-infra` |
 
 A wrong `standalone.control_repository_path` is caught at runtime — by the self-update
@@ -791,12 +794,13 @@ wait for before systemd escalates to SIGKILL
 *Standalone only. In a fleet, your Puppet server (or existing apply pipeline) already
 provides continuous convergence.*
 
-With `standalone.self_update.manage` on, the module installs and keeps converged the full
-self-update loop: `gitlab-runner-apply.service` + `.timer` (fetch,
+With `standalone.manage` on, the module installs `gitlab-runner-healthcheck.service` + `.timer`
+(liveness). With `standalone.self_update.manage` also on, it installs and keeps converged the
+self-update loop, `gitlab-runner-apply.service` + `.timer` (fetch,
 verify the commit signature, reset to the remote branch, apply through the apply script above,
-default every 5 minutes), and
-`gitlab-runner-healthcheck.service` + `.timer`. Nothing needs to be copied or enabled by hand;
-the timers are started and enabled by the apply that installs them.
+default every 5 minutes), and layers the loop-supervision checks into the healthcheck. Nothing
+needs to be copied or enabled by hand; the timers are started and enabled by the apply that
+installs them.
 
 systemd serialises runs (a oneshot service never overlaps itself, so no external locking is
 needed), and `SuccessExitStatus=2` treats Puppet's "changes applied" exit code as success, so only
@@ -902,7 +906,7 @@ Confirm nothing is in a failed state:
 systemctl list-units --failed
 ```
 
-Confirm the apply and healthcheck timers are scheduled (with the self-update toggle on):
+Confirm the timers are scheduled (the healthcheck timer with `standalone.manage`; the apply timer with the self-update loop):
 
 ```
 systemctl list-timers 'gitlab-runner-*'
@@ -927,10 +931,11 @@ gitlab-runner verify
 
 [`gitlab-runner verify`](https://docs.gitlab.com/runner/commands/#gitlab-runner-verify) asks
 GitLab whether each registered runner can connect; run it by hand after rotating tokens. The
-module's healthcheck timer does not run `verify`; it checks the manager service, the rootless
-Docker daemon (as the runner user), that the apply timer is still enabled and armed, and checkout
-staleness against the remote, surfacing failures as failed units. Token validity is the one check
-you make manually.
+module's healthcheck timer does not run `verify`; it checks the manager service and the rootless
+Docker daemon (as the runner user) on any standalone host, and — where the self-update loop is
+enabled — also that the apply timer is still enabled and armed, checkout staleness against the
+remote, and the bootstrap gems, surfacing failures as failed units. Token validity is the one
+check you make manually.
 
 On the GitLab Runners page, a connected runner turns green and is shown online. A runner always
 appears once created, so it is that online state, not its mere presence, that confirms the host

REFERENCE.md

@@ -16,7 +16,7 @@
 * `rootless_gitlab_runner::config`: Renders the runner configuration, the no-detach-netns drop-in and the secret-store directory.
 * `rootless_gitlab_runner::packages`: Installs the packages listed under `packages.install`.
 * `rootless_gitlab_runner::rootless_docker`: Brings up the rootless docker user daemon behind a fail-loud preflight.
-* `rootless_gitlab_runner::self_update`: Installs the standalone apply script and the optional self-update loop.
+* `rootless_gitlab_runner::self_update`: Installs the standalone apply script and liveness healthcheck, plus the optional self-update loop.
 * `rootless_gitlab_runner::service`: Manages the runner system service and its privilege-drop drop-in.
 * `rootless_gitlab_runner::user`: Owns the runner group, user and home.
 
@@ -355,8 +355,10 @@ optional self-update loop nested inside.
 Options:
 
 * **:manage** `Boolean`: Declares the host standalone: installs the apply script
-(`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition
-of the apply command for timer-driven and manual runs. Default false.
+(`/usr/local/sbin/rootless-gitlab-runner-apply`, the single definition of
+the apply command for timer-driven and manual runs) and the liveness
+healthcheck (manager service, rootless docker daemon), whether or not the
+self-update loop is enabled. Default false.
 * **:control_repository_path** `Stdlib::Absolutepath`: Root-owned checkout of the control repository on the host (the apply and
 self-update target). The apply's manifest, module directory, Hiera
 configuration and optional `Puppetfile` derive strictly from the documented
@@ -373,16 +375,18 @@ prepended to the self-update service's PATH so the timer-driven,
 non-login apply can find them. Default `/opt/puppetlabs/bin`; the
 AIO-reuse topology sets `/opt/puppetlabs/puppet/bin` (gem executables
 live there, not in the symlink farm).
-* **:healthcheck_interval** `String[1]`: systemd time span between healthcheck runs. Default `15min`.
+* **:healthcheck_interval** `String[1]`: systemd time span between liveness healthcheck runs. Default `15min`.
 * **:self_update** `Struct[{ manage => Boolean, apply_interval => String[1], apply_timeout => String[1] }]`: The self-update loop: `manage` (default false) installs a oneshot
 service + timer that fetch the control repository, verify the commit
 signature, reset to the remote branch, run `r10k puppetfile install` and
-re-apply — plus the healthcheck script + timer. Only valid on a
-standalone host: enabling it with `standalone.manage` off fails at
-compile time. Never enable it where a Puppet server or r10k already
-deploys the host. `apply_interval` (default `5min`) is the timer's
-cadence; `apply_timeout` (default `15min`) the apply service's
-`TimeoutStartSec=` (a oneshot unit has no default start timeout).
+re-apply. It also layers the loop-supervision checks (apply-timer state,
+checkout staleness, bootstrap-gem presence) into the healthcheck script,
+which itself installs with `standalone.manage`. Only valid on a standalone
+host: enabling it with `standalone.manage` off fails at compile time.
+Never enable it where a Puppet server or r10k already deploys the host.
+`apply_interval` (default `5min`) is the timer's cadence; `apply_timeout`
+(default `15min`) the apply service's `TimeoutStartSec=` (a oneshot unit
+has no default start timeout).
 
 ## Functions
 

manifests/init.pp

@@ -195,8 +195,10 @@
 #   optional self-update loop nested inside.
 # @option standalone [Boolean] :manage
 #   Declares the host standalone: installs the apply script
-#   (`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition
-#   of the apply command for timer-driven and manual runs. Default false.
+#   (`/usr/local/sbin/rootless-gitlab-runner-apply`, the single definition of
+#   the apply command for timer-driven and manual runs) and the liveness
+#   healthcheck (manager service, rootless docker daemon), whether or not the
+#   self-update loop is enabled. Default false.
 # @option standalone [Stdlib::Absolutepath] :control_repository_path
 #   Root-owned checkout of the control repository on the host (the apply and
 #   self-update target). The apply's manifest, module directory, Hiera
@@ -219,17 +221,19 @@
 #   AIO-reuse topology sets `/opt/puppetlabs/puppet/bin` (gem executables
 #   live there, not in the symlink farm).
 # @option standalone [String[1]] :healthcheck_interval
-#   systemd time span between healthcheck runs. Default `15min`.
+#   systemd time span between liveness healthcheck runs. Default `15min`.
 # @option standalone [Struct[{ manage => Boolean, apply_interval => String[1], apply_timeout => String[1] }]] :self_update
 #   The self-update loop: `manage` (default false) installs a oneshot
 #   service + timer that fetch the control repository, verify the commit
 #   signature, reset to the remote branch, run `r10k puppetfile install` and
-#   re-apply — plus the healthcheck script + timer. Only valid on a
-#   standalone host: enabling it with `standalone.manage` off fails at
-#   compile time. Never enable it where a Puppet server or r10k already
-#   deploys the host. `apply_interval` (default `5min`) is the timer's
-#   cadence; `apply_timeout` (default `15min`) the apply service's
-#   `TimeoutStartSec=` (a oneshot unit has no default start timeout).
+#   re-apply. It also layers the loop-supervision checks (apply-timer state,
+#   checkout staleness, bootstrap-gem presence) into the healthcheck script,
+#   which itself installs with `standalone.manage`. Only valid on a standalone
+#   host: enabling it with `standalone.manage` off fails at compile time.
+#   Never enable it where a Puppet server or r10k already deploys the host.
+#   `apply_interval` (default `5min`) is the timer's cadence; `apply_timeout`
+#   (default `15min`) the apply service's `TimeoutStartSec=` (a oneshot unit
+#   has no default start timeout).
 class rootless_gitlab_runner (
   Integer[1]                       $concurrent,
   Array[Hash]                      $runners,
@@ -290,12 +294,13 @@
   }
 
   # The uid is host data with no sensible module default. Fail loud where it
-  # is needed but unset, instead of inventing one. The self-update loop needs
-  # it too: its healthcheck probes the rootless daemon as the runner user.
-  if $runner_account['uid'] =~ Undef and ($runner_account['manage'] or $rootless_docker['manage'] or $standalone['self_update']['manage']) {
+  # is needed but unset, instead of inventing one. A standalone host needs it
+  # too: the liveness healthcheck probes the rootless daemon as the runner
+  # user, so the uid is required wherever the healthcheck installs.
+  if $runner_account['uid'] =~ Undef and ($runner_account['manage'] or $rootless_docker['manage'] or $standalone['manage']) {
     fail(join([
       'rootless_gitlab_runner: runner_account.uid must be set when runner_account.manage, ',
-      'rootless_docker.manage or standalone.self_update.manage is enabled',
+      'rootless_docker.manage or standalone.manage is enabled',
     ]))
   }