Prevent MSRs from leaking across calls to `MultiuseSandbox::restore` by ludfjig · Pull Request #991 · hyperlight-dev/hyperlight

ludfjig · 2025-10-27T19:36:32Z

The goal is to prevent guest state to persist across snapshot-restores.

Kvm: directly disallows read/writes to msrs using KVM_X86_SET_MSR_FILTER. The guest will trap immediately and error will be returned
Hyperv (mshv+whp): The read/writes are allowed (no way to prevent it (intercept exists but doesn't cover all msrs)). Instead restore will reset set msrs using a hardcoded list of MSRs that are known to be writeable. The baseline values of these MSRs are captured on vm-creation. A limitation is that any MSR that is guest-writable and not in this list, has the potential to leak.

Future: When scrub-parition hvcall is ready, we will switch to it, which is guaranteed reset all MSRs. (won't exist for kvm though, which is fine)

~~Will mark ready for review once KVM releases new version, which should include newly added KVM_X86_SET_MSR_FILTER vm ioctl that this PR depends on, see rust-vmm/kvm#359~~

Copilot

Pull request overview

This PR prevents x86_64 model-specific register (MSR) state from persisting across snapshot restores by adding explicit MSR capture/reset semantics to snapshots, enforcing an MSR allow-list contract, and (on KVM) denying guest MSR access by default via an MSR filter.

Changes:

Capture MSR reset state into Snapshot (and snapshot-on-disk format), and restore it during MultiUseSandbox::from_snapshot / MultiUseSandbox::restore.
Add SandboxConfiguration::allow_msrs (bounded allow list) and enforce allow-list compatibility on restore (destination must be a superset).
Implement backend-specific MSR handling: KVM deny-by-default filtering + violation reporting; MSHV/WHP MSR read/write support for reset.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tests/rust_guests/simpleguest/src/main.rs	Adds guest-side RDMSR/WRMSR and SWAPGS helpers for host MSR tests.
src/hyperlight_host/src/sandbox/snapshot/mod.rs	Extends `Snapshot` with optional captured MSRs and snapshot allow list.
src/hyperlight_host/src/sandbox/snapshot/file/mod.rs	Persists/loads MSR fields and validates `msrs`/`allowed_msrs` consistency.
src/hyperlight_host/src/sandbox/snapshot/file/config.rs	Adds MSR fields to OCI snapshot config and serde tests/schema pin updates.
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs	Adds end-to-end disk snapshot tests for MSR capture/compat/allow-list rules.
src/hyperlight_host/src/sandbox/initialized_multi_use.rs	Captures MSRs on snapshot and restores them on `from_snapshot`/`restore` + extensive MSR tests.
src/hyperlight_host/src/sandbox/config.rs	Introduces MSR allow-list storage and capacity-checked `allow_msrs`.
src/hyperlight_host/src/mem/mgr.rs	Threads MSR state through snapshot construction.
src/hyperlight_host/src/hypervisor/virtual_machine/whp.rs	Adds WHP MSR register mapping + get/set MSRs for reset.
src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs	Adds MSHV MSR mapping + get/set MSRs for reset.
src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs	Extends `VirtualMachine` with MSR read/write APIs + validation helpers/errors.
src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs	Implements KVM MSR filter configuration, filtered-exit handling, and KVM MSR get/set.
src/hyperlight_host/src/hypervisor/regs/x86_64/special_regs.rs	Adds test asserting x2APIC stays disabled by default.
src/hyperlight_host/src/hypervisor/regs/x86_64/msrs.rs	New MSR reset table + snapshot validation logic and related constants/tests.
src/hyperlight_host/src/hypervisor/regs/x86_64/mod.rs	Exposes the new MSR module via `pub(crate) use`.
src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs	Captures baseline MSR reset state at VM creation and implements restore logic.
src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs	Wires KVM MSR violations into `RunVmError` and stores `msr_reset` in `HyperlightVm`.
src/hyperlight_host/src/error.rs	Adds `HyperlightError::{MsrReadViolation, MsrWriteViolation}` and marks them poisoning on KVM.
Justfile	Adds ignored host-dependent MSR audit test recipes to `test-isolated`.
docs/msr.md	Documents MSR reset set, backend differences, validation rules, and limitations.
CHANGELOG.md	Adds breaking-change entry for MSR allow-list + restore behavior.
.github/workflows/ValidatePullRequest.yml	Disables matrix `fail-fast` for PR validation workflow.

KVM denies guest MSR access by default. SandboxConfiguration::allow_msrs permits selected MSRs. MSHV and WHP have no per-MSR filter. Hyperlight captures exposed retained MSR state at VM creation and resets it on restore. Captured MSR state persists in OCI snapshots. KVM denials report the MSR index. Unsupported accesses on MSHV and WHP raise a guest general protection fault. Both failures poison the sandbox. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

syntactically

This looks functionally quite good! I left a number of review comments inline, which are mostly about improving clarity / things whose function was not obvious to me.

Beyond what I've explicitly called out in the review comments, I think the high level structure of both the code and the documentation could be a little bit clearer. For the code, I felt like it was not always clear what was HV-specific and what wasn't, and there was just generally a lot of jumping around to follow e.g. the initialisation sequences and compare kvm vs mshv for things like allowlist filtering. For the documentation, there are currently a lot of sections that are all level 2 headings, some of which are related to each other and some of which are not, and it feels like it jumps between different things a few times, making it harder to follow; probably there is some way to use heading structure to organise it a bit more nicely) could be a bit clearer.

syntactically · 2026-07-21T17:28:58Z

+`EFER`, `APIC_BASE`, `FS_BASE`, and `GS_BASE` belong to the special-register
+state.
+
+The two halves are established differently. Resolution checks the


I don't follow this paragraph. Is "Resolution" a specific process? If so, what? What are the "two halves"? Why does "the host-wrtiable half" "hold by construction"? What is mshv doing in here?

syntactically · 2026-07-21T17:30:30Z

+guest-controlled state and are not reset. Each row below names the guest state
+that persists.
+
+| MSR (index) | Retained guest state |


Do we need to regularly cross-check this table against updates to the hypervisor?

syntactically · 2026-07-21T17:33:19Z

+
+* The snapshot's allow list must be a subset of the destination's. A
+  destination that allows at least as much accepts the snapshot.
+* Every supplied index must belong to the destination reset set.


On mshv, are the "supplied index"es in the snapshot "every msr that mshv intercepts" or "just the ones specifically allowed"? If the former, isn't this overly-restrictive? If the latter, isn't this enforced by the previous bullet as well? I assume it's an invariant (independent of snapshotting) that every allowlisted msr is in the reset set.

syntactically · 2026-07-21T17:33:50Z

+capturing sandbox's allow list. `validate_snapshot` enforces two rules against
+the destination VM's reset set:
+
+* The snapshot's allow list must be a subset of the destination's. A


Why not reinitialise the allow list from the one in the snapshot (changing the filters if necessary)?

syntactically · 2026-07-21T17:36:52Z

+
+## KVM
+
+KVM installs a deny filter over the full MSR space. Allowed indices form the


You probably already checked this but I just wanted to call out the need to be sure that KVM doesn't have any special handling for certain things (I could imagine syscall vectors or something) that bypasses the filter.

syntactically · 2026-07-21T19:06:31Z

        {
            use mshv_bindings::MSHV_PT_BIT_LAPIC;
-            pr.pt_flags = 1u64 << MSHV_PT_BIT_LAPIC;
+            pr.pt_flags |= 1u64 << MSHV_PT_BIT_LAPIC;


Is this change preserving any specific other flag?

syntactically · 2026-07-21T19:14:18Z

+// PRED_CMD (0x49) and FLUSH_CMD (0x10B) are intentionally absent: they are
+// write-only commands with no retained state, so they cannot be reset and
+// therefore cannot be allowed.
+const MSR_TABLE: &[u32] = &[


Does this need to include the x2APIC accesses for mshv guests which I presume can write APIC_BASE?

syntactically · 2026-07-21T19:17:27Z

                })?;
+
+            // Restore captured MSR state.
+            #[cfg(target_arch = "x86_64")]


It would be nice if most of this x86_64-specific stuff could be factored out into e.g. arch::handle_arch_specific_state.

syntactically · 2026-07-21T19:23:55Z

+                } else if kernel_gs_uses_instruction_side_effect && index == KERNEL_GS_BASE {
+                    // Direct WRMSR is denied. The dedicated SWAPGS test proves
+                    // the instruction-side mutation is restored.
+                } else if (index == 0x10 && !kernel_gs_uses_instruction_side_effect)


Why is kernel_gs_uses_instruction_side_effect relevant to TSC?

Also, for the MSRs which have named constants in regs/x86_64/msrs.rs, it would be nice to use the constants in the tests?

syntactically · 2026-07-21T19:25:05Z

+                    || matches!(index, 0xE7 | 0xE8)
+                {
+                    assert_guest_counter_is_writable_and_restored(&mut sbox, index);
+                } else if let Some(sentinel) = positive_write_sentinel(index) {


What does "positive_write_sentinel" mean?

ludfjig force-pushed the reset2 branch 7 times, most recently from aeca04d to 968f2f4 Compare October 31, 2025 00:29

ludfjig force-pushed the reset2 branch from 968f2f4 to 91ba9e0 Compare December 2, 2025 00:27

ludfjig added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Dec 2, 2025

ludfjig force-pushed the reset2 branch 3 times, most recently from e859ff9 to e793129 Compare December 8, 2025 18:42

ludfjig changed the title ~~Reset more state when restoring snapshot~~ Reset MSRs when restoring snapshot Dec 18, 2025

ludfjig force-pushed the reset2 branch from e793129 to 68751e4 Compare February 10, 2026 22:17

ludfjig changed the title ~~Reset MSRs when restoring snapshot~~ Disallow MSR read/writes Feb 10, 2026

ludfjig force-pushed the reset2 branch 5 times, most recently from 8f55249 to 3bb2294 Compare February 13, 2026 23:16

jsturtevant mentioned this pull request Feb 18, 2026

Ensure that system registers don't leak information between uses of a sandbox #791

Open

ludfjig force-pushed the reset2 branch 2 times, most recently from 032168d to c0383dd Compare February 18, 2026 19:27

ludfjig force-pushed the reset2 branch from c0383dd to a84fa74 Compare March 16, 2026 23:04

ludfjig force-pushed the reset2 branch 6 times, most recently from 2e98073 to 089227c Compare July 11, 2026 07:10

ludfjig force-pushed the reset2 branch 10 times, most recently from 99c2a45 to a4cf1de Compare July 17, 2026 04:06

ludfjig changed the title ~~Disallow MSR read/writes~~ Prevent MSRs from leaking across calls to MultiuseSandbox::restore Jul 17, 2026

ludfjig marked this pull request as ready for review July 17, 2026 04:12

Copilot AI review requested due to automatic review settings July 17, 2026 04:12

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners July 17, 2026 04:12

ludfjig force-pushed the reset2 branch from a4cf1de to ca728b5 Compare July 17, 2026 04:13

Copilot started reviewing on behalf of ludfjig July 17, 2026 04:15 View session

Copilot AI reviewed Jul 17, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/snapshot/mod.rs Outdated

Comment thread docs/msr.md Outdated

Comment thread CHANGELOG.md Outdated

Comment thread src/hyperlight_host/src/sandbox/snapshot/file/config.rs

Comment thread src/hyperlight_host/src/sandbox/snapshot/file/config.rs

ludfjig force-pushed the reset2 branch from ca728b5 to c079263 Compare July 17, 2026 04:42

ludfjig force-pushed the reset2 branch from c079263 to 50b829f Compare July 17, 2026 16:12

ludfjig added the ready-for-review PR is ready for (re-)review label Jul 17, 2026

syntactically reviewed Jul 21, 2026

View reviewed changes


		## KVM

		KVM installs a deny filter over the full MSR space. Allowed indices form the

Uh oh!

Conversation

ludfjig commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

syntactically left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ludfjig commented Oct 27, 2025 •

edited

Loading