diff --git a/CHANGELOG.md b/CHANGELOG.md index 224aa64..20e1683 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ module follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Every subkey ships a module default and the groups merge deep across Hiera layers, so node data holds only its deviations, and a mistyped subkey now fails the compile instead of being silently ignored. - The runner-token store reads as part of the runner family: `tokens` is now `runner_tokens`, alongside `runners` and `runner_defaults`. The automatic `Sensitive` wrap on lookup moves with the new key; the host's secret store file must follow the rename. - The apply script now installs on any declared-standalone host (`standalone.manage: true`), not only with the self-update loop, so manual applies and the loop share the same single apply command; enabling the loop on a host not declared standalone fails at compile time. +- The liveness healthcheck now installs on any declared-standalone host (`standalone.manage: true`), not only with the self-update loop, so a standalone host without the loop still has its manager service and rootless Docker daemon monitored (cadence `standalone.healthcheck_interval`); the loop's own supervision checks (apply-timer state, checkout staleness, bootstrap-gem presence) layer on only when the loop is enabled. - Freshly provisioned hosts now get a 165536-wide subordinate-ID range, which satisfies the ID-mapping requirement of nested rootless BuildKit builds. - Subordinate UID/GID ranges are now provisioned by `rootless_docker.manage`, together with the other rootless-Docker prerequisites, instead of by the runner-account toggle: rootless Docker can now be brought up on a host whose runner account is owned by another system. An existing range entry is never overwritten, and `runner_account.manage` keeps owning the group, user, and home. - The managed runner service's `DOCKER_HOST` is now module-owned and derived from `runner_account.uid`; `runner_service.environment` carries additional variables only and rejects a `DOCKER_HOST` line, and a managed service requires `runner_account.uid`. diff --git a/README.md b/README.md index f6eefd7..2413632 100644 --- a/README.md +++ b/README.md @@ -233,8 +233,8 @@ ownership toggle, opt-in by default: | Runner account (group, user, home) | [`runner_account.manage`](#runner_account) | `false` | | Rootless-Docker daemon bring-up, subordinate IDs | [`rootless_docker.manage`](#rootless_docker) | `false` | | Runner service + its systemd drop-in | [`runner_service.manage`](#runner_service) | `false` | -| Standalone topology (the apply script) | [`standalone.manage`](#standalone) | `false` | -| Self-update loop + healthcheck | [`standalone.self_update.manage`](#standaloneself_update) | `false` | +| Standalone topology (apply script + liveness healthcheck) | [`standalone.manage`](#standalone) | `false` | +| Self-update loop | [`standalone.self_update.manage`](#standaloneself_update) | `false` | The `manage` keys are **persistent ownership switches**, not one-shot bootstrap flags: set once in the host's Hiera and left on, so every apply keeps owning and drift-correcting that @@ -347,7 +347,10 @@ window ([Restarts and graceful shutdown](#restarts-and-graceful-shutdown)). rather than being deployed by a Puppet server. It installs the apply script (`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition of the apply command for manual runs and the self-update loop alike -([Applying the configuration](#applying-the-configuration)). The apply's manifest, module +([Applying the configuration](#applying-the-configuration)), and a liveness healthcheck +(script + timer, default every `15min`, `standalone.healthcheck_interval`) that verifies the +manager service and the rootless daemon (`docker info` as the runner user) — installed on any +standalone host, with or without the loop. The apply's manifest, module directory and Hiera configuration derive from the documented control-repository layout (`puppet/manifests/site.pp`, `puppet/modules`, `puppet/hiera.yaml` beneath the checkout); the isolated Puppet state directories are `standalone.puppet_confdir`/`standalone.puppet_vardir`, @@ -357,18 +360,18 @@ non-login apply. #### `standalone.self_update` With `standalone.self_update.manage` on — only valid on a standalone host: enabling it with -`standalone.manage` off fails at compile time — the module installs the self-update loop, two -units each on its own timer: - -- A oneshot systemd service + timer (default every `5min`, `standalone.self_update.apply_interval`) - that fetch the control-repository checkout, run `git verify-commit` on the remote branch - (only signed commits are applied), reset to it, install Puppetfile-pinned modules via - `r10k puppetfile install` (a no-op without a Puppetfile), and re-apply through the apply - script above. -- A healthcheck script + timer (default every `15min`, `standalone.healthcheck_interval`) that - verifies the manager service, the rootless daemon (`docker info` as the runner user, from a - non-login context), and that the checkout is not stale against the remote (a dead pull - credential fails loud instead of leaving the host applying old code behind a green timer). +`standalone.manage` off fails at compile time — the module installs the self-update loop: a +oneshot systemd service + timer (default every `5min`, `standalone.self_update.apply_interval`) +that fetch the control-repository checkout, run `git verify-commit` on the remote branch (only +signed commits are applied), reset to it, install Puppetfile-pinned modules via +`r10k puppetfile install` (a no-op without a Puppetfile), and re-apply through the apply script +above. + +The loop also layers its own supervision onto the liveness healthcheck that `standalone.manage` +installs: it adds checks that the apply timer is enabled and armed, that the checkout is not +stale against the remote (a dead pull credential fails loud instead of leaving the host applying +old code behind a green timer), and that the bootstrap gems (`r10k`, `hiera-eyaml`) are present +in the AIO Ruby. The service sets `HOME=/root` (git/SSH need it) and an explicit `TimeoutStartSec` (`standalone.self_update.apply_timeout`); Puppet exit code 2 ("changes applied") counts as @@ -386,7 +389,7 @@ host-specific: | Hiera key | Description | Default | |---|---|---| -| `runner_account.uid` | Numeric uid of the runner account; the rootless runtime paths (`/run/user/`, the docker socket) derive from it | none — required when `runner_account.manage`, `rootless_docker.manage` or `standalone.self_update.manage` is on, or for a `socket_mount` runner | +| `runner_account.uid` | Numeric uid of the runner account; the rootless runtime paths (`/run/user/`, the docker socket) derive from it | none — required when `runner_account.manage`, `rootless_docker.manage` or `standalone.manage` is on, or for a `socket_mount` runner | | `standalone.control_repository_path` | Checkout of the control repository on the host (the apply and self-update target) | `/opt/gitlab-runner-infra` | A wrong `standalone.control_repository_path` is caught at runtime — by the self-update @@ -791,12 +794,13 @@ wait for before systemd escalates to SIGKILL *Standalone only. In a fleet, your Puppet server (or existing apply pipeline) already provides continuous convergence.* -With `standalone.self_update.manage` on, the module installs and keeps converged the full -self-update loop: `gitlab-runner-apply.service` + `.timer` (fetch, +With `standalone.manage` on, the module installs `gitlab-runner-healthcheck.service` + `.timer` +(liveness). With `standalone.self_update.manage` also on, it installs and keeps converged the +self-update loop, `gitlab-runner-apply.service` + `.timer` (fetch, verify the commit signature, reset to the remote branch, apply through the apply script above, -default every 5 minutes), and -`gitlab-runner-healthcheck.service` + `.timer`. Nothing needs to be copied or enabled by hand; -the timers are started and enabled by the apply that installs them. +default every 5 minutes), and layers the loop-supervision checks into the healthcheck. Nothing +needs to be copied or enabled by hand; the timers are started and enabled by the apply that +installs them. systemd serialises runs (a oneshot service never overlaps itself, so no external locking is needed), and `SuccessExitStatus=2` treats Puppet's "changes applied" exit code as success, so only @@ -902,7 +906,7 @@ Confirm nothing is in a failed state: systemctl list-units --failed ``` -Confirm the apply and healthcheck timers are scheduled (with the self-update toggle on): +Confirm the timers are scheduled (the healthcheck timer with `standalone.manage`; the apply timer with the self-update loop): ``` systemctl list-timers 'gitlab-runner-*' @@ -927,10 +931,11 @@ gitlab-runner verify [`gitlab-runner verify`](https://docs.gitlab.com/runner/commands/#gitlab-runner-verify) asks GitLab whether each registered runner can connect; run it by hand after rotating tokens. The -module's healthcheck timer does not run `verify`; it checks the manager service, the rootless -Docker daemon (as the runner user), that the apply timer is still enabled and armed, and checkout -staleness against the remote, surfacing failures as failed units. Token validity is the one check -you make manually. +module's healthcheck timer does not run `verify`; it checks the manager service and the rootless +Docker daemon (as the runner user) on any standalone host, and — where the self-update loop is +enabled — also that the apply timer is still enabled and armed, checkout staleness against the +remote, and the bootstrap gems, surfacing failures as failed units. Token validity is the one +check you make manually. On the GitLab Runners page, a connected runner turns green and is shown online. A runner always appears once created, so it is that online state, not its mere presence, that confirms the host diff --git a/REFERENCE.md b/REFERENCE.md index 10597a3..4fb3994 100644 --- a/REFERENCE.md +++ b/REFERENCE.md @@ -16,7 +16,7 @@ * `rootless_gitlab_runner::config`: Renders the runner configuration, the no-detach-netns drop-in and the secret-store directory. * `rootless_gitlab_runner::packages`: Installs the packages listed under `packages.install`. * `rootless_gitlab_runner::rootless_docker`: Brings up the rootless docker user daemon behind a fail-loud preflight. -* `rootless_gitlab_runner::self_update`: Installs the standalone apply script and the optional self-update loop. +* `rootless_gitlab_runner::self_update`: Installs the standalone apply script and liveness healthcheck, plus the optional self-update loop. * `rootless_gitlab_runner::service`: Manages the runner system service and its privilege-drop drop-in. * `rootless_gitlab_runner::user`: Owns the runner group, user and home. @@ -355,8 +355,10 @@ optional self-update loop nested inside. Options: * **:manage** `Boolean`: Declares the host standalone: installs the apply script -(`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition -of the apply command for timer-driven and manual runs. Default false. +(`/usr/local/sbin/rootless-gitlab-runner-apply`, the single definition of +the apply command for timer-driven and manual runs) and the liveness +healthcheck (manager service, rootless docker daemon), whether or not the +self-update loop is enabled. Default false. * **:control_repository_path** `Stdlib::Absolutepath`: Root-owned checkout of the control repository on the host (the apply and self-update target). The apply's manifest, module directory, Hiera configuration and optional `Puppetfile` derive strictly from the documented @@ -373,16 +375,18 @@ prepended to the self-update service's PATH so the timer-driven, non-login apply can find them. Default `/opt/puppetlabs/bin`; the AIO-reuse topology sets `/opt/puppetlabs/puppet/bin` (gem executables live there, not in the symlink farm). -* **:healthcheck_interval** `String[1]`: systemd time span between healthcheck runs. Default `15min`. +* **:healthcheck_interval** `String[1]`: systemd time span between liveness healthcheck runs. Default `15min`. * **:self_update** `Struct[{ manage => Boolean, apply_interval => String[1], apply_timeout => String[1] }]`: The self-update loop: `manage` (default false) installs a oneshot service + timer that fetch the control repository, verify the commit signature, reset to the remote branch, run `r10k puppetfile install` and -re-apply — plus the healthcheck script + timer. Only valid on a -standalone host: enabling it with `standalone.manage` off fails at -compile time. Never enable it where a Puppet server or r10k already -deploys the host. `apply_interval` (default `5min`) is the timer's -cadence; `apply_timeout` (default `15min`) the apply service's -`TimeoutStartSec=` (a oneshot unit has no default start timeout). +re-apply. It also layers the loop-supervision checks (apply-timer state, +checkout staleness, bootstrap-gem presence) into the healthcheck script, +which itself installs with `standalone.manage`. Only valid on a standalone +host: enabling it with `standalone.manage` off fails at compile time. +Never enable it where a Puppet server or r10k already deploys the host. +`apply_interval` (default `5min`) is the timer's cadence; `apply_timeout` +(default `15min`) the apply service's `TimeoutStartSec=` (a oneshot unit +has no default start timeout). ## Functions diff --git a/manifests/init.pp b/manifests/init.pp index 381c57a..95ea755 100644 --- a/manifests/init.pp +++ b/manifests/init.pp @@ -195,8 +195,10 @@ # optional self-update loop nested inside. # @option standalone [Boolean] :manage # Declares the host standalone: installs the apply script -# (`/usr/local/sbin/rootless-gitlab-runner-apply`), the single definition -# of the apply command for timer-driven and manual runs. Default false. +# (`/usr/local/sbin/rootless-gitlab-runner-apply`, the single definition of +# the apply command for timer-driven and manual runs) and the liveness +# healthcheck (manager service, rootless docker daemon), whether or not the +# self-update loop is enabled. Default false. # @option standalone [Stdlib::Absolutepath] :control_repository_path # Root-owned checkout of the control repository on the host (the apply and # self-update target). The apply's manifest, module directory, Hiera @@ -219,17 +221,19 @@ # AIO-reuse topology sets `/opt/puppetlabs/puppet/bin` (gem executables # live there, not in the symlink farm). # @option standalone [String[1]] :healthcheck_interval -# systemd time span between healthcheck runs. Default `15min`. +# systemd time span between liveness healthcheck runs. Default `15min`. # @option standalone [Struct[{ manage => Boolean, apply_interval => String[1], apply_timeout => String[1] }]] :self_update # The self-update loop: `manage` (default false) installs a oneshot # service + timer that fetch the control repository, verify the commit # signature, reset to the remote branch, run `r10k puppetfile install` and -# re-apply — plus the healthcheck script + timer. Only valid on a -# standalone host: enabling it with `standalone.manage` off fails at -# compile time. Never enable it where a Puppet server or r10k already -# deploys the host. `apply_interval` (default `5min`) is the timer's -# cadence; `apply_timeout` (default `15min`) the apply service's -# `TimeoutStartSec=` (a oneshot unit has no default start timeout). +# re-apply. It also layers the loop-supervision checks (apply-timer state, +# checkout staleness, bootstrap-gem presence) into the healthcheck script, +# which itself installs with `standalone.manage`. Only valid on a standalone +# host: enabling it with `standalone.manage` off fails at compile time. +# Never enable it where a Puppet server or r10k already deploys the host. +# `apply_interval` (default `5min`) is the timer's cadence; `apply_timeout` +# (default `15min`) the apply service's `TimeoutStartSec=` (a oneshot unit +# has no default start timeout). class rootless_gitlab_runner ( Integer[1] $concurrent, Array[Hash] $runners, @@ -290,12 +294,13 @@ } # The uid is host data with no sensible module default. Fail loud where it - # is needed but unset, instead of inventing one. The self-update loop needs - # it too: its healthcheck probes the rootless daemon as the runner user. - if $runner_account['uid'] =~ Undef and ($runner_account['manage'] or $rootless_docker['manage'] or $standalone['self_update']['manage']) { + # is needed but unset, instead of inventing one. A standalone host needs it + # too: the liveness healthcheck probes the rootless daemon as the runner + # user, so the uid is required wherever the healthcheck installs. + if $runner_account['uid'] =~ Undef and ($runner_account['manage'] or $rootless_docker['manage'] or $standalone['manage']) { fail(join([ 'rootless_gitlab_runner: runner_account.uid must be set when runner_account.manage, ', - 'rootless_docker.manage or standalone.self_update.manage is enabled', + 'rootless_docker.manage or standalone.manage is enabled', ])) } diff --git a/manifests/self_update.pp b/manifests/self_update.pp index c49e6eb..3a00e69 100644 --- a/manifests/self_update.pp +++ b/manifests/self_update.pp @@ -1,16 +1,33 @@ -# @summary Installs the standalone apply script and the optional self-update loop. +# @summary Installs the standalone apply script and liveness healthcheck, plus the optional self-update loop. # @api private class rootless_gitlab_runner::self_update { assert_private() $standalone = $rootless_gitlab_runner::standalone $apply_script = '/usr/local/sbin/rootless-gitlab-runner-apply' + $healthcheck_script = '/usr/local/sbin/rootless-gitlab-runner-healthcheck' $control_repository_path = $standalone['control_repository_path'] - # The apply script installs on any standalone host, not only with the - # self-update loop: it is the single definition of the apply command for - # manual runs and for the loop alike. + # The apply script and the liveness healthcheck install on any standalone + # host, not only with the self-update loop: the apply script is the single + # definition of the apply command for manual runs and the loop alike, and + # liveness (manager service, rootless daemon) is meaningful whether or not + # an automatic loop drives convergence. The loop-supervision checks layer + # into the healthcheck script only when self_update.manage is on (rendered + # in the template). if $standalone['manage'] { + # This class's own reload point; refreshonly. Deliberately separate from + # service.pp's exec: sharing one would order it both before the runner + # service (its subscriber) and after these unit files (its notifiers) — + # a dependency cycle under the service -> self_update class ordering. The + # apply units below (gated on self_update.manage) notify it too, which is + # sound because self_update.manage implies standalone.manage. + exec { 'rootless_gitlab_runner daemon-reload (standalone)': + command => 'systemctl daemon-reload', + path => ['/usr/bin', '/bin'], + refreshonly => true, + } + # The manifest, module directory and Hiera configuration derive strictly # from the documented control-repository layout — the layout is the # contract, not a parameter. @@ -28,20 +45,6 @@ 'vardir' => $standalone['puppet_vardir'], }), } - } - - if $standalone['self_update']['manage'] { - $healthcheck_script = '/usr/local/sbin/rootless-gitlab-runner-healthcheck' - - # This class's own reload point; refreshonly. Deliberately separate from - # service.pp's exec: sharing one would order it both before the runner - # service (its subscriber) and after these unit files (its notifiers) — - # a dependency cycle under the service -> self_update class ordering. - exec { 'rootless_gitlab_runner daemon-reload (self-update)': - command => 'systemctl daemon-reload', - path => ['/usr/bin', '/bin'], - refreshonly => true, - } file { $healthcheck_script: ensure => file, @@ -55,76 +58,81 @@ 'control_repository_path' => $control_repository_path, 'control_repository_branch' => $standalone['control_repository_branch'], 'service_name' => $rootless_gitlab_runner::service_name, + 'self_update_manage' => $standalone['self_update']['manage'], }), } - file { '/etc/systemd/system/gitlab-runner-apply.service': + file { '/etc/systemd/system/gitlab-runner-healthcheck.service': ensure => file, owner => 'root', group => 'root', mode => '0644', - content => epp('rootless_gitlab_runner/apply.service.epp', { - 'apply_script' => $apply_script, - 'control_repository_path' => $control_repository_path, - 'control_repository_branch' => $standalone['control_repository_branch'], - 'apply_timeout' => $standalone['self_update']['apply_timeout'], - 'puppet_bindir' => $standalone['puppet_bindir'], + content => epp('rootless_gitlab_runner/healthcheck.service.epp', { + 'healthcheck_script' => $healthcheck_script, }), - notify => Exec['rootless_gitlab_runner daemon-reload (self-update)'], + notify => Exec['rootless_gitlab_runner daemon-reload (standalone)'], } - file { '/etc/systemd/system/gitlab-runner-apply.timer': + file { '/etc/systemd/system/gitlab-runner-healthcheck.timer': ensure => file, owner => 'root', group => 'root', mode => '0644', - content => epp('rootless_gitlab_runner/apply.timer.epp', { - 'apply_interval' => $standalone['self_update']['apply_interval'], + content => epp('rootless_gitlab_runner/healthcheck.timer.epp', { + 'healthcheck_interval' => $standalone['healthcheck_interval'], }), - notify => Exec['rootless_gitlab_runner daemon-reload (self-update)'], + notify => Exec['rootless_gitlab_runner daemon-reload (standalone)'], } - file { '/etc/systemd/system/gitlab-runner-healthcheck.service': + # The timer subscribes to its unit files (via the daemon-reload exec), so a + # changed interval takes effect in the same apply, not the next boot. + service { 'gitlab-runner-healthcheck.timer': + ensure => running, + enable => true, + subscribe => [ + File['/etc/systemd/system/gitlab-runner-healthcheck.service'], + File['/etc/systemd/system/gitlab-runner-healthcheck.timer'], + Exec['rootless_gitlab_runner daemon-reload (standalone)'], + ], + } + } + + if $standalone['self_update']['manage'] { + file { '/etc/systemd/system/gitlab-runner-apply.service': ensure => file, owner => 'root', group => 'root', mode => '0644', - content => epp('rootless_gitlab_runner/healthcheck.service.epp', { - 'healthcheck_script' => $healthcheck_script, + content => epp('rootless_gitlab_runner/apply.service.epp', { + 'apply_script' => $apply_script, + 'control_repository_path' => $control_repository_path, + 'control_repository_branch' => $standalone['control_repository_branch'], + 'apply_timeout' => $standalone['self_update']['apply_timeout'], + 'puppet_bindir' => $standalone['puppet_bindir'], }), - notify => Exec['rootless_gitlab_runner daemon-reload (self-update)'], + notify => Exec['rootless_gitlab_runner daemon-reload (standalone)'], } - file { '/etc/systemd/system/gitlab-runner-healthcheck.timer': + file { '/etc/systemd/system/gitlab-runner-apply.timer': ensure => file, owner => 'root', group => 'root', mode => '0644', - content => epp('rootless_gitlab_runner/healthcheck.timer.epp', { - 'healthcheck_interval' => $standalone['healthcheck_interval'], + content => epp('rootless_gitlab_runner/apply.timer.epp', { + 'apply_interval' => $standalone['self_update']['apply_interval'], }), - notify => Exec['rootless_gitlab_runner daemon-reload (self-update)'], + notify => Exec['rootless_gitlab_runner daemon-reload (standalone)'], } - # The timers subscribe to their unit files (via the daemon-reload exec), - # so a changed interval takes effect in the same apply, not the next boot. + # The timer subscribes to its unit files (via the daemon-reload exec), so a + # changed interval takes effect in the same apply, not the next boot. service { 'gitlab-runner-apply.timer': ensure => running, enable => true, subscribe => [ File['/etc/systemd/system/gitlab-runner-apply.service'], File['/etc/systemd/system/gitlab-runner-apply.timer'], - Exec['rootless_gitlab_runner daemon-reload (self-update)'], - ], - } - - service { 'gitlab-runner-healthcheck.timer': - ensure => running, - enable => true, - subscribe => [ - File['/etc/systemd/system/gitlab-runner-healthcheck.service'], - File['/etc/systemd/system/gitlab-runner-healthcheck.timer'], - Exec['rootless_gitlab_runner daemon-reload (self-update)'], + Exec['rootless_gitlab_runner daemon-reload (standalone)'], ], } } diff --git a/spec/classes/rootless_gitlab_runner_spec.rb b/spec/classes/rootless_gitlab_runner_spec.rb index 577186a..674ce54 100644 --- a/spec/classes/rootless_gitlab_runner_spec.rb +++ b/spec/classes/rootless_gitlab_runner_spec.rb @@ -71,6 +71,15 @@ def account_with_uid(uid, extra = {}) end end + it 'installs nothing from the standalone layer on a fleet host (no healthcheck, no apply)' do + is_expected.not_to contain_file('/usr/local/sbin/rootless-gitlab-runner-apply') + is_expected.not_to contain_file('/usr/local/sbin/rootless-gitlab-runner-healthcheck') + is_expected.not_to contain_file('/etc/systemd/system/gitlab-runner-healthcheck.service') + is_expected.not_to contain_file('/etc/systemd/system/gitlab-runner-healthcheck.timer') + is_expected.not_to contain_service('gitlab-runner-healthcheck.timer') + is_expected.not_to contain_exec('rootless_gitlab_runner daemon-reload (standalone)') + end + it 'chains the classes so a one-shot fresh apply converges in order' do is_expected.to contain_class('rootless_gitlab_runner::apt_repos').that_comes_before('Class[rootless_gitlab_runner::packages]') is_expected.to contain_class('rootless_gitlab_runner::packages').that_comes_before('Class[rootless_gitlab_runner::user]') @@ -403,7 +412,7 @@ def account_with_uid(uid, extra = {}) { 'runner_account.manage' => { 'runner_account' => { 'manage' => true } }, 'rootless_docker.manage' => { 'rootless_docker' => { 'manage' => true } }, - 'self_update.manage' => { 'standalone' => { 'manage' => true, 'self_update' => { 'manage' => true } } }, + 'standalone.manage' => { 'standalone' => { 'manage' => true } }, }.each do |toggle, overrides| context "with #{toggle} enabled" do let(:params) { overrides.to_h { |name, over| [name, struct_param(name, over)] } } @@ -810,8 +819,9 @@ def account_with_uid(uid, extra = {}) context 'with standalone.manage alone (no self-update loop)' do let(:params) do - { 'standalone' => struct_param('standalone', 'manage' => true, - 'control_repository_path' => '/opt/infra') } + { 'standalone' => struct_param('standalone', 'manage' => true, + 'control_repository_path' => '/opt/infra'), + 'runner_account' => account_with_uid(4242) } end it { is_expected.to compile.with_all_deps } @@ -830,12 +840,30 @@ def account_with_uid(uid, extra = {}) .with_content(%r{"/opt/infra/puppet/manifests/site\.pp"}) end - it 'installs no self-update units and no healthcheck without the loop' do - is_expected.not_to contain_file('/usr/local/sbin/rootless-gitlab-runner-healthcheck') + it 'installs the liveness healthcheck (script, unit, timer) even without the loop' do + is_expected.to contain_file('/usr/local/sbin/rootless-gitlab-runner-healthcheck').with_mode('0755') + is_expected.to contain_file('/etc/systemd/system/gitlab-runner-healthcheck.service') + is_expected.to contain_file('/etc/systemd/system/gitlab-runner-healthcheck.timer') + is_expected.to contain_service('gitlab-runner-healthcheck.timer') + .with('ensure' => 'running', 'enable' => true) + is_expected.to contain_exec('rootless_gitlab_runner daemon-reload (standalone)') + end + + it 'renders only the liveness layer: no apply-timer, staleness, or gem checks' do + is_expected.to contain_file('/usr/local/sbin/rootless-gitlab-runner-healthcheck') + .with_content(%r{is-active --quiet 'gitlab-runner'}) + .with_content(%r{rootless docker user unit is not active}) + .with_content(%r{docker info}) + .without_content(%r{gitlab-runner-apply\.timer}) + .without_content(%r{ls-remote}) + .without_content(%r{checkout is stale}) + .without_content(%r{gem list --installed}) + end + + it 'installs no self-update apply units without the loop' do is_expected.not_to contain_file('/etc/systemd/system/gitlab-runner-apply.service') is_expected.not_to contain_file('/etc/systemd/system/gitlab-runner-apply.timer') is_expected.not_to contain_service('gitlab-runner-apply.timer') - is_expected.not_to contain_exec('rootless_gitlab_runner daemon-reload (self-update)') end end @@ -893,7 +921,7 @@ def account_with_uid(uid, extra = {}) .without_content(%r{OnFailure}) end - it 'renders the healthcheck with daemon probe, apply-timer and SHA-staleness assertions' do + it 'renders the healthcheck with daemon probe, apply-timer, SHA-staleness and bootstrap-gem assertions' do is_expected.to contain_file('/usr/local/sbin/rootless-gitlab-runner-healthcheck') .with_mode('0755') .with_content(%r{docker info}) @@ -903,6 +931,9 @@ def account_with_uid(uid, extra = {}) .with_content(%r{is-active --quiet gitlab-runner-apply\.timer}) .with_content(%r{ls-remote origin 'refs/heads/main'}) .with_content(%r{checkout is stale}) + .with_content(%r{for gem in r10k hiera-eyaml}) + .with_content(%r{/opt/puppetlabs/puppet/bin/gem list --installed --exact}) + .with_content(%r{bootstrap gem '\$\{gem\}' is missing from the AIO Ruby}) end %w[gitlab-runner-apply.timer gitlab-runner-healthcheck.timer].each do |t| diff --git a/spec/fixtures/golden/healthcheck.sh.golden b/spec/fixtures/golden/healthcheck.sh.golden index 042554d..c5a3a4f 100644 --- a/spec/fixtures/golden/healthcheck.sh.golden +++ b/spec/fixtures/golden/healthcheck.sh.golden @@ -4,10 +4,10 @@ # is overwritten on the next apply. # # Health check: silent when healthy, loud on failure (non-zero exit + message -# on stderr, which lands in the journal). Checks the manager service, the -# rootless docker daemon (as the runner user, from a non-login context), and -# checkout staleness — a dead pull credential must not leave the host applying -# stale main behind a green apply timer. +# on stderr, which lands in the journal). Two layers: liveness (manager +# service, rootless docker daemon as the runner user) runs on any standalone +# host; the self-update-loop supervision (apply timer, checkout staleness, +# bootstrap gems) is rendered only where the loop drives convergence. set -uo pipefail status=0 @@ -24,9 +24,19 @@ as_runner() { "$@" } +# ── Liveness: meaningful on any standalone host, loop or not. ── systemctl is-active --quiet 'gitlab-runner' \ || err "manager service 'gitlab-runner' is not active" +as_runner systemctl --user is-active --quiet docker \ + || err "rootless docker user unit is not active" + +# setuptool check is prerequisites-only; docker info is the daemon health probe. +as_runner docker info >/dev/null 2>&1 \ + || err "docker info failed as gitlab-runner" + +# ── Self-update-loop supervision: only where an automatic loop promises +# convergence (a manually-applied host is legitimately behind origin). ── # The apply timer drives self-update; a disabled or stopped timer silently # halts convergence and would otherwise surface only later as checkout # staleness. Assert it is both enabled (survives reboot) and armed now. @@ -35,13 +45,6 @@ systemctl is-enabled --quiet gitlab-runner-apply.timer \ systemctl is-active --quiet gitlab-runner-apply.timer \ || err "apply timer 'gitlab-runner-apply.timer' is not active (self-update halted)" -as_runner systemctl --user is-active --quiet docker \ - || err "rootless docker user unit is not active" - -# setuptool check is prerequisites-only; docker info is the daemon health probe. -as_runner docker info >/dev/null 2>&1 \ - || err "docker info failed as gitlab-runner" - # SHA staleness (mandatory): compare the local checkout against the remote # head without mutating the checkout. An unreachable origin is itself a # failure — that is exactly the dead-credential case. @@ -54,4 +57,14 @@ elif [[ "${local_sha}" != "${remote_sha}" ]]; then err "checkout is stale: local ${local_sha} != origin/main ${remote_sha}" fi +# Bootstrap gems in the AIO Ruby: r10k and hiera-eyaml. A missing gem also +# fails the apply timer (r10k before puppet apply, hiera-eyaml at lookup); +# asserting them here turns that into one clear message instead of a raw +# apply failure. The AIO gem lives at a fixed path, not the puppet_bindir +# symlink farm. +for gem in r10k hiera-eyaml; do + /opt/puppetlabs/puppet/bin/gem list --installed --exact "${gem}" >/dev/null 2>&1 \ + || err "bootstrap gem '${gem}' is missing from the AIO Ruby (self-update apply will fail)" +done + exit "${status}" diff --git a/templates/healthcheck.sh.epp b/templates/healthcheck.sh.epp index 547f395..9399934 100644 --- a/templates/healthcheck.sh.epp +++ b/templates/healthcheck.sh.epp @@ -4,6 +4,7 @@ String $control_repository_path, String $control_repository_branch, String $service_name, + Boolean $self_update_manage, | -%> <%# The manager-service check below asserts the packaged gitlab-runner systemd unit: $service_name is the unit name the gitlab-runner package defines, @@ -14,10 +15,10 @@ # is overwritten on the next apply. # # Health check: silent when healthy, loud on failure (non-zero exit + message -# on stderr, which lands in the journal). Checks the manager service, the -# rootless docker daemon (as the runner user, from a non-login context), and -# checkout staleness — a dead pull credential must not leave the host applying -# stale <%= $control_repository_branch %> behind a green apply timer. +# on stderr, which lands in the journal). Two layers: liveness (manager +# service, rootless docker daemon as the runner user) runs on any standalone +# host; the self-update-loop supervision (apply timer, checkout staleness, +# bootstrap gems) is rendered only where the loop drives convergence. set -uo pipefail status=0 @@ -34,9 +35,20 @@ as_runner() { "$@" } +# ── Liveness: meaningful on any standalone host, loop or not. ── systemctl is-active --quiet '<%= $service_name %>' \ || err "manager service '<%= $service_name %>' is not active" +as_runner systemctl --user is-active --quiet docker \ + || err "rootless docker user unit is not active" + +# setuptool check is prerequisites-only; docker info is the daemon health probe. +as_runner docker info >/dev/null 2>&1 \ + || err "docker info failed as <%= $runner_name %>" +<% if $self_update_manage { -%> + +# ── Self-update-loop supervision: only where an automatic loop promises +# convergence (a manually-applied host is legitimately behind origin). ── # The apply timer drives self-update; a disabled or stopped timer silently # halts convergence and would otherwise surface only later as checkout # staleness. Assert it is both enabled (survives reboot) and armed now. @@ -45,13 +57,6 @@ systemctl is-enabled --quiet gitlab-runner-apply.timer \ systemctl is-active --quiet gitlab-runner-apply.timer \ || err "apply timer 'gitlab-runner-apply.timer' is not active (self-update halted)" -as_runner systemctl --user is-active --quiet docker \ - || err "rootless docker user unit is not active" - -# setuptool check is prerequisites-only; docker info is the daemon health probe. -as_runner docker info >/dev/null 2>&1 \ - || err "docker info failed as <%= $runner_name %>" - # SHA staleness (mandatory): compare the local checkout against the remote # head without mutating the checkout. An unreachable origin is itself a # failure — that is exactly the dead-credential case. @@ -64,4 +69,15 @@ elif [[ "${local_sha}" != "${remote_sha}" ]]; then err "checkout is stale: local ${local_sha} != origin/<%= $control_repository_branch %> ${remote_sha}" fi +# Bootstrap gems in the AIO Ruby: r10k and hiera-eyaml. A missing gem also +# fails the apply timer (r10k before puppet apply, hiera-eyaml at lookup); +# asserting them here turns that into one clear message instead of a raw +# apply failure. The AIO gem lives at a fixed path, not the puppet_bindir +# symlink farm. +for gem in r10k hiera-eyaml; do + /opt/puppetlabs/puppet/bin/gem list --installed --exact "${gem}" >/dev/null 2>&1 \ + || err "bootstrap gem '${gem}' is missing from the AIO Ruby (self-update apply will fail)" +done +<% } -%> + exit "${status}"