Skip to content

Harden PCP systemd units with minimal, service-only changes#2655

Open
kurik wants to merge 5 commits into
performancecopilot:mainfrom
kurik:pcp-systemd-hardening
Open

Harden PCP systemd units with minimal, service-only changes#2655
kurik wants to merge 5 commits into
performancecopilot:mainfrom
kurik:pcp-systemd-hardening

Conversation

@kurik

@kurik kurik commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Add least-privilege directives to unit files only; omit SystemCallFilter, ProtectHostname on pmcd, and PrivateTmp on farm services to avoid performance impact and preserve existing behavior.

Patch description

This change hardens PCP systemd service units for issue #2644
(#2644) by adding common least-privilege directives
(ProtectHome, ProtectClock, ProtectKernel*, RestrictNamespaces, UMask=0027, etc.) to the main PCP services
and their timer/check/farm companions.

The patch is intentionally minimal in scope: it touches only systemd unit template files (*.service.in) and
packaging metadata. There are no changes to application code, libraries, PMDAs, or rc scripts.
SystemCallFilter is deliberately not used, based on upstream discussion that syscall filtering caused large
QA slowdowns (e.g. qa/966 going from ~17s to ~89s).

Omitted hardening options

Two options that would otherwise be straightforward to add were left out after test-suite analysis:

ProtectHostname=yes on pmcd.service

In normal production use, this is unlikely to matter much: pmcd reads the hostname at startup and most
deployments do not change the system hostname at runtime.

It does break hostname-following behavior that PCP relies on when the hostname changes. pmcd detects changes
via gethostname() and signals clients with PMCD_HOSTNAME_CHANGE; pmlogger and pmie then restart to pick up
the new name. With ProtectHostname=yes, pmcd sees a frozen hostname namespace and never propagates that
change.

Test impact: qa/1441 failed — after a hostname change, the primary pmlogger never restarted and still
referenced the old hostname.

Status: Omitted from pmcd.service with an inline comment. This is a reasonable hardening candidate later, but
would need either a different pmcd hostname-detection approach or test rework.

PrivateTmp=yes on pmie_farm.service and pmlogger_farm.service

For typical deployments, farm instances write logs under /var/log/pcp/... from control files, so PrivateTmp
may have little practical effect.

It does break non-primary farm instances that use log paths outside the service’s private /tmp — including
user-configured locations and QA tests that deliberately use /tmp/<test-pid>/....

Test impact: qa/1060 failed — the non-primary pmie instance could not create its nested log tree under
/tmp/1060-*.

Status: Omitted from both farm service units with an inline comment. Low-hanging fruit for a future hardening
pass, but farm log paths are user-configurable, so proper validation would need more than a quick unit-file
change.

What was added

Hardening was applied across pmcd, pmproxy, pmlogger, pmie, pmfind, atop, geolocate, pmseries, and their
check/daily/farm companion units, with service-specific exceptions documented in comments (e.g.
NoNewPrivileges omitted on pmcd because it drops to the pcp user via setuid; NotifyAccess=main on pmproxy).

Rationale for deferring ProtectHostname and PrivateTmp

Both options are attractive from a systemd-analyze security perspective and would likely be fine in many
production setups. They were not included in this iteration because:

  1. They have clear functional interactions with hostname following and farm log paths.
  2. Failures showed up only in targeted QA tests, not in obvious production breakage — but that is not
    enough confidence without broader validation.
  3. Properly addressing them would require additional testing and possible QA/test rework, which is out of
    scope for this minimal, time-constrained patch.

The goal here is a small, low-risk hardening step that improves exposure scores without touching runtime code
or risking the performance regressions seen with heavier sandboxing. ProtectHostname and PrivateTmp remain
documented candidates for a follow-up once there is time to validate behavior and adjust tests as needed.

Add least-privilege directives to unit files only; omit SystemCallFilter,
ProtectHostname on pmcd, and PrivateTmp on farm services to avoid
performance impact and preserve existing behavior.
@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 393bd212-602a-4f03-a513-fdd28bd480b4

📥 Commits

Reviewing files that changed from the base of the PR and between 4115ed2 and 5db9224.

📒 Files selected for processing (1)
  • src/pmcd/pmcd.service.in
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/pmcd/pmcd.service.in

📝 Walkthrough

Summary by CodeRabbit

  • Security
    • Strengthened systemd sandboxing and hardening across PCP monitoring, logging, analysis, proxy, geolocation, geolocation import, series import, and related services.
    • Added protections against privilege escalation and unsafe temp/key handling (private temp where applicable and private keyrings), plus stronger isolation of clock/hostname/kernel interfaces, kernel logs/modules, control groups, and tunables.
    • Tightened runtime behavior with native syscall architecture; services intentionally avoid syscall filtering and omit UMask where needed to preserve existing file-permission behavior.
  • Reliability
    • Improved service readiness/notification reporting for the proxy by adjusting notification scope to the main process.

Walkthrough

Systemd service templates across PCP components now add sandboxing, privilege, namespace, kernel-interface, keyring, and syscall-architecture restrictions. pmproxy also narrows notification access to the main process, while comments document intentional exceptions.

Changes

Systemd service hardening

Layer / File(s) Summary
Core service hardening
src/pcp/atop/*, src/pcp/geolocate/*, src/pmcd/*, src/pmfind/*, src/pmproxy/*, src/pmseries/*
Core PCP services gain systemd sandboxing, protected kernel interfaces, namespace restrictions, private keyrings, native syscall architectures, and documented configuration exceptions. pmproxy changes NotifyAccess from all to main.
PMIE service hardening
src/pmie/*
PMIE service variants gain matching hardening directives, with comments documenting omitted SystemCallFilter, PrivateTmp, NoNewPrivileges, and UMask settings where applicable.
PML logger service hardening
src/pmlogger/*
PML logger service variants gain sandboxing, privilege, IPC, namespace, keyring, and syscall-architecture restrictions, with documented exceptions for farm paths and file permissions.

Possibly related issues

Poem

A rabbit hops through units bright,
Guarding kernels through the night.
Private keys and paths secure,
Service boundaries now endure.
“No syscall filter,” whispers hare—
Hardened services everywhere!

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: hardening PCP systemd units with limited service-file updates.
Description check ✅ Passed The description is related to the patch and discusses the same systemd hardening work, despite some detail mismatches.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/pcp/geolocate/pcp-geolocate.service.in`:
- Around line 12-27: Harden all three units by adding UMask=0027 immediately
after KeyringMode=private in src/pcp/geolocate/pcp-geolocate.service.in (lines
12-27), src/pmproxy/pmproxy_check.service.in (lines 14-30), and
src/pmproxy/pmproxy_daily.service.in (lines 14-30); no other service settings
need to change.

In `@src/pmie/pmie_check.service.in`:
- Around line 17-32: Add UMask=0027 to the hardening settings in
src/pmie/pmie_check.service.in lines 17-32 and src/pmie/pmie_daily.service.in
lines 16-32, matching the existing PMIE units; only document an exception
instead if either service intentionally requires different file-creation
permissions.

In `@src/pmie/pmie_farm_check.service.in`:
- Around line 16-32: Document the intentional omission of RemoveIPC=yes in the
hardening settings near the existing NoNewPrivileges comment, briefly explaining
the root-running service’s IPC cleanup requirement if applicable. Do not add the
directive; only add a concise rationale consistent with the surrounding
comments.

In `@src/pmie/pmie.service.in`:
- Around line 23-38: Document the intentional omission of ProtectHostname and
PrivateTmp in pmie.service.in by adding a brief inline comment near the related
hardening directives. State the required runtime behavior, such as
hostname-change detection or /tmp access for inference temporary files, and
follow the exception-documentation style used by the other service units.

In `@src/pmproxy/pmproxy.service.in`:
- Around line 14-30: Document the intentional omission of PrivateTmp in the
hardening settings for src/pmproxy/pmproxy.service.in lines 14-30 and
src/pmcd/pmcd.service.in lines 14-30 by adding brief inline comments explaining
why shared /tmp access is required, matching the existing
exception-documentation style; no other service settings need changes.
- Around line 14-30: Remove the RemoveIPC=yes directive from the pmproxy systemd
unit hardening settings, leaving the surrounding protections unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Repository UI (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: e33c0695-89fe-4107-afa9-623997cb6a7d

📥 Commits

Reviewing files that changed from the base of the PR and between 97ec201 and 2753eec.

📒 Files selected for processing (19)
  • src/pcp/atop/atop-daily.service
  • src/pcp/atop/atop.service.in
  • src/pcp/geolocate/pcp-geolocate.service.in
  • src/pmcd/pmcd.service.in
  • src/pmfind/pmfind.service.in
  • src/pmie/pmie.service.in
  • src/pmie/pmie_check.service.in
  • src/pmie/pmie_daily.service.in
  • src/pmie/pmie_farm.service.in
  • src/pmie/pmie_farm_check.service.in
  • src/pmlogger/pmlogger.service.in
  • src/pmlogger/pmlogger_check.service.in
  • src/pmlogger/pmlogger_daily.service.in
  • src/pmlogger/pmlogger_farm.service.in
  • src/pmlogger/pmlogger_farm_check.service.in
  • src/pmproxy/pmproxy.service.in
  • src/pmproxy/pmproxy_check.service.in
  • src/pmproxy/pmproxy_daily.service.in
  • src/pmseries/pmseries_import.service.in

Comment thread src/pcp/geolocate/pcp-geolocate.service.in
Comment thread src/pmie/pmie_check.service.in
Comment thread src/pmie/pmie_farm_check.service.in Outdated
Comment on lines +16 to +32
# Hardening: no SystemCallFilter (performance impact).
# NoNewPrivileges omitted because pmiectl uses runaspcp (setuid).
PrivateTmp=yes
ProtectHome=yes
ProtectClock=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictNamespaces=yes
SystemCallArchitectures=native
KeyringMode=private
UMask=0027

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the omission of RemoveIPC=yes.

All other PMIE service units in this PR include RemoveIPC=yes, but pmie_farm_check.service.in omits it without comment. If this is intentional (e.g., the root-running service must not strip IPC objects on exit), add a brief comment explaining why, consistent with the documented NoNewPrivileges exception.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/pmie/pmie_farm_check.service.in` around lines 16 - 32, Document the
intentional omission of RemoveIPC=yes in the hardening settings near the
existing NoNewPrivileges comment, briefly explaining the root-running service’s
IPC cleanup requirement if applicable. Do not add the directive; only add a
concise rationale consistent with the surrounding comments.

Comment thread src/pmie/pmie.service.in Outdated
Comment on lines +23 to +38
# Hardening: no SystemCallFilter (performance impact).
ProtectHome=yes
NoNewPrivileges=yes
RemoveIPC=yes
ProtectClock=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
RestrictNamespaces=yes
SystemCallArchitectures=native
KeyringMode=private
UMask=0027

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Document the omission of ProtectHostname and PrivateTmp.

All other PMIE service units in this PR include ProtectHostname=yes, and most include PrivateTmp=yes (or document its absence, as pmie_farm.service.in does). pmie.service.in omits both without any inline comment explaining why. If these omissions are intentional (e.g., pmie needs hostname-change detection like pmcd.service, or needs /tmp access for inference temp files), add a brief comment for consistency with the documented exceptions in the other units.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/pmie/pmie.service.in` around lines 23 - 38, Document the intentional
omission of ProtectHostname and PrivateTmp in pmie.service.in by adding a brief
inline comment near the related hardening directives. State the required runtime
behavior, such as hostname-change detection or /tmp access for inference
temporary files, and follow the exception-documentation style used by the other
service units.

Comment thread src/pmproxy/pmproxy.service.in Outdated
kurik added 2 commits July 13, 2026 21:01
Leave the default umask so PMDA Install/make artifacts stay
world-readable/executable; UMask=0027 broke sample PMDA and CI QA.
Drop ProtectKernelModules and RestrictNamespaces from pmcd.service so
the dm PMDA can ioctl /dev/mapper/control and run dmsetup; fixes qa/359
on Ubuntu 20.04/22.04 CI.
@kmcdonell

Copy link
Copy Markdown
Member

@kurik are you proposing this for the 7.1.6 release?
I'm nervous about this change late in the release cycle, so NOT 🤠 or 🤡 this time.

Do you have empirical data to show that there's no significant performance impact here (the syscall assertion in issue #2644 is speculation, I don't have any data (yet) to prove this is the root cause of the slowdown I observed in my parallel efforts down this path).

Of course, I have no visibility into any firm customer-pressure to address this quickly which would be a non-technical counter argument. 😄

@kurik

kurik commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@kurik are you proposing this for the 7.1.6 release? I'm nervous about this change late in the release cycle, so NOT 🤠 or 🤡 this time.

No, I am not proposing this for 7.1.6 release. It still might need some work as there are some differences on Debian (as I see in the CI).

Do you have empirical data to show that there's no significant performance impact here (the syscall assertion in issue #2644 is speculation, I don't have any data (yet) to prove this is the root cause of the slowdown I observed in my parallel efforts down this path).

I do not have the data anymore, but I can easily generate it. I was using check -TT to get timestamps and then I compared the duration for each test. The result was +/- 1s per test, while the overall duration was 2s faster when run with the hardening. So it is IMO within natural fluctuation. I will keep the data from the next run when I investigate the Debian failures.

Of course, I have no visibility into any firm customer-pressure to address this quickly which would be a non-technical counter argument. 😄

There is no pressure on my side.

@kmcdonell

Copy link
Copy Markdown
Member

Excellent @kurik, let's try and nail this down as early as possible in the next release cycle. So I don't forget, tasks I can see are:

  1. Compare my service changes with yours and focus on the differences
  2. I think I have QA fixes for most of the PrivateTmp fallout ... this is only needed for QA AFAICT and we can work around it there, so I suggest that one can go back in the mix
  3. Some controlled timing experiments to see what the overhead really is.
  4. Determine if the final exposure score for all the services is acceptable.

kurik added 2 commits July 15, 2026 07:21
* Removed "RemoveIPC" from all PCP's unit files to avoid unexpected data
lost
* Tweaked pmcd.service to ensure proper work of dm PMDA
@kurik

kurik commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Hi @kmcdonell ,
finally I have some minimal set of systemd hardening options, which work even on Ubuntu :-) ( I typically do not test on Ubuntu, so this takes me more time... ).

As part of this exercise I tried to compare performance with and without this hardening. At first, I run all tests in the local group without the hardening and considered this as a baseline. Then I did two runs of the same test set with the hardening enabled. Here is the outcome:

Average performance impact

  • Mean per-test delta: +0.026 s (effectively zero at 1 s resolution)

Largest single spike (not reproducible):

  • Test 1108: 1 s → 52 s in hardening run 2 only (+51 s). Run 1 was 1 s. This is IMO some pmlogextract/cron noise, not a hardening effect.

Largest reproducible slowdown (both hardened runs):

  • Test 1845: 1 s → ~10.5 s (+9.5 s) — pmlogger_daily for pmproxy-pushed archives; the only pmproxy-related case with a consistent delta
  • Test 1662: 19 s → 22.5 s (+3.5 s) — pmproxy valgrind leak test
  • Six tests total reproducibly slower: +21.5 s combined

Reproducible speedups (both hardened runs faster):

  • Tests 061 (−24.5 s), 1614 (−18.5 s, failed→passed), 1626 (−11.5 s) — net −54.5 s on those three.

Conclusion:

The data does not support a measurable suite-wide performance cost. The only plausibly hardening-related reproducible case is 1845.

One more comment:

I haven't yet investigated why the success rate is higher with hardening enabled compared to the baseline run. I know some tests are unstable, but this still warrants a closer look when I have some spare time:

Run Passed Failed Not run
No hardening 1,492 14 193
Hardening 1 1,497 11 191
Hardening 2 1,501 7 191

@kurik

kurik commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Compare my service changes with yours and focus on the differences

That would be great. What I learned during this exercise is the fact there might be some side effects one (at least me :-) ) do not expect. Specifically I mean RemoveIPC when the removal is based on UID/GID, so if you have two services (i.e. pmcd and pmie) where only one service uses IPC (the pmcd) then restart of the second service (pmie) removes all IPC artifacts owned by the same UID/GID (shared memory, semaphores, ...) just because it is run as the same UID/GID as the first service.

As such, I definitely agree with you to merge these changes early in the release cycle, so there is as much time as possible to catch possible issues.

I think I have QA fixes for most of the PrivateTmp fallout ... this is only needed for QA AFAICT and we can work around it there, so I suggest that one can go back in the mix

This is great. I tried to come up with a minimal set of hardening that wouldn't even affect the test suite. However, it seems that settings like PrivateTmp don't affect individual services; they only affect specific tests.

Some controlled timing experiments to see what the overhead really is.
Determine if the final exposure score for all the services is acceptable.

I am not sure how much we can have some controlled environment when nowadays is "everything" virtual, even the hardware :-)
I am typically using TestingFarm for my test environment, which offers semi-transparent access to AWS (for x86_64) or IBM Cloud (s390x) etc. My point of view would be rather to have more test runs with a tested feature disabled and then with the feature enabled and do some statistic on top of this (averages, spikes, the most significant deviations, etc.) However as you have access to bare metal (bozo machines?) we can do both.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants