vm: mmap wake pays first-touch faults from the VM's quota — dense hosts must trade memory density against wake latency

[CMGS]

Summary

#187 correctly moved provisioning outside the child's ceiling: Prepare(..., deferQuota=true) holds cpu.max=max through the paused window and Arm installs the finite quota after the memory load, before resume. That fixes the copy-mode half.

#186 already called the mmap half out and consciously left it as a doc note:

Option 1 also covers mmap's deferred first-touch cost only if migration happens after warm-up, which it can't reasonably — so for mmap the residual is inherent and worth a doc note either way.

The note duly landed in docs/vm.md:

The mmap restore mode's deferred first-touch faults are inherent guest-lifetime work and do pay the VM's quota — on dense hosts prefer copy mode for latency-critical wakes or grant quota headroom.

This issue argues the residual is worth more than a caveat, because it binds hardest in exactly the regime the cgroup work was built to enable, and because our current measurements structurally cannot see it.

Why the caveat is load-bearing rather than cosmetic

Restore mode and quota are chosen for opposite reasons, and both point at the same hosts:

• mmap is what makes dense hosts affordable — sibling clones of one golden share clean pages through the host page cache, so memory stops scaling with VM count.
• A deliberately low quota is what makes dense hosts affordable on CPU — a Burstable shape (--cpu 2 --cpu-weight 25, per docs/vm.md) lets low-priority VMs yield under contention.

Take both and the wake path is throttled by the very knob that made the density possible. The documented escape hatches each surrender one of the two goals:

• copy mode → gives up the page-cache sharing that motivated mmap in the first place;
• "grant quota headroom" → gives up the CPU ceiling, i.e. the tail bound the scope exists to provide;
• --cpu-burst-us → the kernel requires burst <= quota (docs/cli.md), so at a low quota the burst credit is small by construction. It cannot fund a first-touch storm.

So on an oversubscribed host there is currently no setting that keeps memory density, a bounded CPU tail, and a fast wake at the same time.

Why our benchmarks do not show it

Arm fires before resume, so the cost lands after the clone/restore call returns. Every clone-latency number we publish — including #186's table, where mmap under the 20-VM storm looks healthy at 114/198/230 ms p50/P90/max against copy's 516/1032/1348 — stops the clock before the guest has faulted in its working set. mmap looking good there is partly the measurement boundary, not only the mechanism.

The user-visible metric is claim to useful work, and that window is precisely where the deferred faults are throttled. I have not measured this — that is the first thing this issue asks for.

Proposal

1. Measure it. Extend the clone bench past the return: claim -> first exec -> steady state, under the #186 storm shape, mmap x {Guaranteed-at-N, Burstable low-weight, small explicit quota}, with nr_throttled / throttled_usec sampled from cpu.stat across the window. That table decides whether anything below is worth building.

2. If it is real: stage the quota instead of arming it once. Arm already establishes the primitive — a scope whose ceiling changes at a lifecycle point, idempotent, with the guest never running uncapped. The gap is that it is a single step-up-to-final. A wake-window policy would arm an elevated ceiling at resume and step down to the configured quota once the working set is resident, gated on a signal rather than a timer where possible:

• simplest: elapsed time (--cpu-wake-quota-us + --cpu-wake-ms), trivially bounded, crude;
• better: step down when cpu.stat throttling stops rising, or when the guest agent reports ready;
• the step-down must be as convergent as Arm is today — a missed transition has to fail closed to the configured quota, never leave a VM permanently uncapped.

3. Cheap partial, independent of the above. copy vs mmap is currently an operator decision made once per clone. Given #187 exempts provisioning from the ceiling, copy's cost now lands in the exempt window while mmap's lands in the charged one — an inversion worth documenting explicitly in the restore_mode guidance, because the right default for a dense host may now differ from the right default for a quiet one.

Notes

• Not urgent for Guaranteed-at-N at generous --cpu, where the quota is not binding during warm-up. It is specifically a dense/oversubscribed-host issue.
• Related: vm: per-VM cgroup v2 CPU scope (request / limit / burst) — vCPU count only bounds guest parallelism #182 (scope), vm: cpuset fence — reserve host cores from the VM population, optional per-VM core placement #184/vm: cpuset fence — reserve host cores from the VM population (#184) #185 (fence), clone/restore execution pays the child VM's cgroup quota — provisioning work throttled under load #186/vm: exempt clone/restore provisioning from the child VM's CPU ceiling (#186) #187 (provisioning exemption).

[CMGS]

Re-read the code to check the "Arm already gives half the primitive" claim above. It holds, and the tree turns out to already contain the hazard a step-down would hit — worth recording before anyone implements this.

cpu.max and cpu.max.burst are order-dependent, and the two directions need opposite orders.

armQuota writes quota then burst:

func armQuota(dir string, k Knobs) error {
	if err := writeControl(dir, maxName, fmt.Sprintf("%d %d", k.QuotaUs, k.PeriodUs)); err != nil {
		return err
	}
	if k.BurstUs > 0 {
		if err := writeControl(dir, burstName, strconv.FormatInt(k.BurstUs, 10)); err != nil {

That is the correct step-up order: raise the ceiling first, then set a burst that is <= the new quota. Validate enforces the same invariant at config time (--cpu-burst-us must be 0..quota).

A step-down inverts it, and Prepare already documents why — from the scope-reuse path:

A reused scope may hold a leftover burst > target quota, which blocks the cpu.max write (kernel requires burst <= quota): zero it first. ENOENT tolerated — pre-5.14 kernels lack the file.

So the wake-window step-down is not a fresh problem: lowering the ceiling from the elevated wake quota to the configured one has to zero (or lower) cpu.max.burst before writing cpu.max, exactly as scope reuse does. An implementation should factor that ordering out of Prepare rather than re-derive it — it is the kind of asymmetry that passes on a quiet host and fails the first time someone combines --cpu-burst-us with a staged wake.

Two consequences for the proposal:

1. It is cheaper than framed above — the primitive is armQuota plus a lowerQuota sibling with the reverse write order, and Arm's idempotence/convergence contract carries over unchanged.
2. It reinforces the fail-closed requirement: a step-down that dies between the burst write and the quota write leaves the VM at the elevated quota with a reduced burst — bounded, but not the configured policy. Convergence needs to re-run the whole pair, not resume mid-sequence.

None of this changes the ask: measure first (proposal 1). Just noting the mechanism is closer to hand than the issue text implies, and where the sharp edge is.

[CMGS]

First P1 attempt (testbed, 985a070, 20-VM storm, golden with a 1.5G tmpfs working set, mmap clones, claim→in-guest-full-scan timed):

shape	clone	scan wall	throttled_usec
Guaranteed-at-2	44–92ms	2.0–2.8s	0
quota=1 core	51–168ms	2.6–3.0s	~0
quota=0.5 core, weight=25	26–59ms	2.2–2.4s	1.16–1.68s

Two findings and two method lessons rather than a verdict:

1. The qualitative claim holds: the 0.5-core shape accumulates 1.2s+ of throttling during the wake window — first-touch work demonstrably pays the VM's quota, and the mmap clone window itself stays thin (26–168ms) exactly as the measurement-boundary argument predicted.
2. Wall time did not degrade — because on this host the scan is dominated by fault-I/O wait (~550–680MB/s effective, i.e. reading the snapshot file), which pays no quota. The CPU share the quota throttles is hidden under the I/O share.
3. Method lesson: shape-sequential runs are poisoned by page-cache warm-up (the third shape ran 15–25% faster from cache heat alone — rerunning shape 1 hot reproduced the 'improvement'). A real P1 table needs warm-up passes + interleaved shapes.
4. Environment lesson: this testbed currently cannot saturate (scheduler anomaly, ~50%/core ceiling affecting plain host userspace too), so quota-binding regimes barely engage. The definitive P1 table should run where the clone/restore execution pays the child VM's cgroup quota — provisioning work throttled under load #186 numbers came from.

Net: the mechanism the issue describes is real (throttle counters prove it) but its wall-clock magnitude on a dense host is still unmeasured. Staging (P2) stays gated on a proper table; the P3 doc inversion note is safe to write regardless.

[CMGS]

Second round on a 384-core host (bd26) that saturates properly (16-core fence measured 160s/10s under the 20-VM storm — the .79 anomaly absent). The results challenge #186's attribution and are worth folding into this issue's P1 design:

Saturated interleaved A/B, copy mode, n=12 each: pre-fix 3542c8a p50/p90 = 418/487ms vs fixed 985a070 = 405/534ms — statistically indistinguishable. The 1032ms pathology did not reproduce here either.

Mechanism probe (8G golden with a 6G resident blob, sampling the clone's scope during the copy window, cold and hot cache):

• clone wall is constant ~3.1s cold and hot;
• the scope consumes < 0.5 core during the window (72–409ms per 2s sample), threads=4, in every run;
• i.e. the copy restore's heavy cost does not run inside the VM's scope at all — it lives in cocoon-side snapshot file preparation (outside the fence and any per-VM quota) and I/O wait, neither of which a scope quota can throttle.

Implication: in this reproduction, the child-scope quota never binds during provisioning, so #187's deferred quota — while semantically right (clean throttle accounting, and protective if CH's restore path ever becomes scope-resident multi-threaded) — shows no measurable latency benefit, and "quota throttles the eager copy" cannot explain the original 530→1032ms doubling.

What would explain it remains open: candidate variables in the original vk measurement that our repro lacks are the real wake path (reseed + networking, vs our --nics 0), a different CH build, peer-transfer-hot page cache with a much larger snapshot, or contention on the caller side rather than the child scope. Suggest the next P1 run happen in the vk environment with cpu.stat sampled from both the child scope and the caller's cgroup — if the throttling shows up on the caller's side, the fix belongs in the vk unit's CPUWeight (docs/vm.md control-plane note), not in cocoon.

(.79 rounds from earlier tonight also relevant: kill-injection and post-resume-ceiling acceptance for #187 both PASS — the correctness half of #187 stands regardless of the perf attribution.)

[CMGS]

Follow-up closing the caller-side hypothesis loop — CMGS pointed out it needs no vk environment: constrain cocoon itself. Same bd26 setup (saturated 16-core fence, 20-VM storm, 2G golden, copy clones ×8 per cell), cocoon run under systemd-run scopes:

caller constraint	p50 / p90 (ms)
bare (root session, unconstrained)	413 / 536
CPUQuota=200%	467 / 596
AllowedCPUs=0-15 (inside the saturated fence)	490 / 545
AllowedCPUs=0-15 + CPUQuota=100%	439 / 571

Even pinning cocoon into the saturated fence at a 1-core quota moves p50 by only +10–20% — no doubling. Combined with the child-scope probe (quota never binds, <0.5 core consumed in-scope), neither side of the cgroup boundary reproduces the 530→1032ms pathology under this repro's shape (--nics 0, 2G golden, NVMe, CH v54 dev).

So the attribution table now reads: child-scope quota — ruled out; caller cgroup — ruled out as a primary cause; remaining candidates are properties of the real vk wake path this repro lacks (networked clone with reseed, the 13–17GB hibernate-snapshot scale, pd-ssd I/O, or the specific CH build). P1's next iteration should instrument that path directly rather than any synthetic. Until then #187 stands as a correctness/accounting fix, and staging (P2) has no measured justification yet.

[CMGS]

Full-scale round per review feedback: same host used at its real proportions — fence 0-382 (383 cores to VMs, core 383 reserved), 384×1-vCPU storm igniting 372 sustained cores (97% saturation), 16G golden with a 12G resident set (vk hibernate scale), latest fork dev CH/FC/FW (5d71c6db / 0033ef64 / 29e5e779), cocoon master vs pre-#187:

mode	PRE p50/p90	FIX p50/p90
copy	2511 / 2955 ms	2522 / 3675 ms
mmap	87 / 96 ms	89 / 134 ms

No defer-quota benefit at production scale either — consistent with every smaller round and with the mechanism probe (the copy cost lives outside the child scope). The 530→1032ms doubling now stands unreproduced across three environments spanning undersaturated, fully-saturated-16-core, and 97%-saturated-383-core regimes with a vk-scale snapshot. At this point the burden shifts to instrumenting the original vk wake path itself; P2 (staged quota) remains without measured justification, P3's doc note stands.

Bycatch worth the round on its own: pinning the control plane to the reserved core surfaced #189 — vCPU validation reads runtime.NumCPU() (process affinity), so a core-pinned operator can't clone multi-vCPU VMs at all, and maxVCPUs silently collapses to 1 on the same path.

[CMGS]

Reviewing the five rounds together, because "we keep not measuring it" has three separate causes and only one of them is about this issue.

1. Four of the five rounds tested #186, not #188

Two different claims have merged in this thread:

	claim	rounds	outcome
#186	provisioning (copy-mode memory load) is throttled by the child scope's quota → 530→1032ms	4	unreproduced across three regimes; the mechanism probe positively showed the copy cost lives outside the child scope
#188	post-resume mmap first-touch faults are throttled → slow wake on dense hosts	1	see below

The saturated A/B, the mechanism probe, the caller-cgroup sweep and the 383-core full-scale round are all #186. That work is not a failed measurement — it is a clean negative result that retires #186's attribution, and it is worth exactly that. It says nothing about #188.

2. The one round that did test #188 succeeded

quota=0.5 core, weight=25 accumulated 1.16–1.68s of throttled_usec during the wake window. The ceiling bound; first-touch work does pay the VM's quota. The mechanism is confirmed. What did not move was wall clock.

3. Why wall clock did not move, stated as the invariant rather than as a host property

CFS bandwidth control charges runnable time. A task blocked in D-state on I/O accrues no runtime and consumes no quota. So:

• cold snapshot file → first touch is a major fault → block on the read → the expensive part of the work costs zero quota. Throttling can only delay the microseconds of CPU on either side of the I/O, and those stalls overlap with I/O the task was going to wait for anyway.
• hot page cache → first touch is a minor fault → no I/O at all → the entire cost is CPU → the quota binds directly with nothing to hide behind.

Round 1's reading ("hidden under the I/O share") is right, but the conclusion drawn from it was that this is a property of that host. It is not — it is a property of cache state.

4. Which means the round measured the one regime where the effect cannot appear

mmap exists so that sibling clones of one golden share its page cache. Clone #1 warms it; clones #2..N are minor faults. The measurement was clone #1.

Read/write compounds it. On MAP_PRIVATE, a read fault against a clean page maps the page-cache page shared (~µs of CPU, the cheapest case); a write fault does CoW — allocate plus a 4 KiB memcpy per page, which is genuinely CPU-bound.

	cold cache	hot cache
read (scan)	major fault, I/O-dominated — what was measured	shared mapping, ~µs CPU
write (dirty)	major fault + CoW	CoW memcpy per page, pure CPU — what a resuming guest does

The test took one corner of that 2x2; production sits in the opposite one.

5. The metric shape also favours hiding it

claim -> in-guest-full-scan over 1.5G is a throughput workload: high fault concurrency, deep pipeline, per-fault stalls overlap. The user-visible metric is claim -> first useful work (first exec returns, or CDP answers) — a latency metric over a small scattered working set, with no pipeline depth for stalls to hide in.

6. The storm is an irrelevant variable here

cpu.max is an absolute ceiling; it throttles on a completely idle host. The 20-VM and 384-VM storms matter for weight arbitration and for the fence — #186's territory — and add nothing to this claim.

So round 1's environment caveat ("this testbed cannot saturate, so quota-binding regimes barely engage") does not apply to #188: quota binding needs no saturation, and the 1.2–1.7s of throttling measured on that very host proves it engaged. That round is stronger evidence than it credited itself with.

7. The missing instrument

Major/minor fault ratio — pgmajfault vs pgfault, or the cgroup's memory.stat. That single number identifies the regime, and therefore whether the effect can appear at all.

Without it a null result is uninterpretable: there is no way to distinguish "no effect" from "wrong regime". None of the five rounds sampled it. Adding it is what makes the existing null results readable.

8. Falsifiable prediction, and the run that settles this

Round 1's own numbers bound the scan's CPU demand between 0.5 and 1 core (throttled at 0.5, ~0 throttle at 1). With the I/O removed, wall time should approach CPU_time / quota:

on a hot cache, quota=0.5 should run ~2x the wall time of quota=2 — visible without statistics.

If a hot-cache, write-touch, latency-metric run still shows flat wall time, #188's premise is wrong and this issue should be closed.

The run:

• one golden, N sibling clones taken sequentially, all mmap;
• discard clone Implement VM networking via CNI with per-VM netns + bridge + tap #1 — its job is to warm the host page cache; measure Replace bridge with TC redirect for VM networking #2..N;
• interleave quota shapes across siblings, not sequentially by shape (round 1's own method lesson);
• working set touched by random write, not sequential read;
• metric claim -> first exec returns, not full-scan wall;
• sample throttled_usec, usage_usec, and the major/minor fault ratio per run.

P2 (staged quota) stays gated on that table. P3's doc note stands either way — and if the prediction holds, the note gets sharper: the copy vs mmap guidance should key on cache state, since cold mmap is I/O-bound and immune to the quota while hot mmap is CPU-bound and fully exposed to it.

[CMGS]

Ran the prescribed run on the 384-core host. The prediction holds — quota=0.5 runs 2.0x the wall time of quota=2, visible without statistics.

Setup per spec: one golden (2 vCPU / 6G, 3G tmpfs working set seeded pre-snapshot), sequential mmap sibling clones, clone #1 discarded as the cache warmer, shapes interleaved T/W/T/W (T = --cpu-quota-us 50000, W = 200000), n=6 each; per run sampled cpu.stat (usage_usec/throttled_usec) on the VM scope and min/maj faults on the VMM process. One deviation: the touch is a sequential full overwrite of the working set (dd conv=notrunc, the guest image has no python3), not random — immaterial in this quadrant since the regime instrument confirms zero major faults, and on hot cache + write every page CoWs exactly once regardless of order (minflt ≈ 800k ≈ 3G/4KiB).

shape	clone ms	claim→first exec ms	touch wall ms (all 6)	usage_us	throttled_us	majflt
T (0.5 core)	58-63	114-154	6650 6806 6824 6859 6867 6923	~3.43M	~3.45M	0
W (2 core)	61-114	115-170	3301 3368 3412 3418 3494 3494	~3.42M	0	0

The arithmetic closes exactly: both shapes burn the same ~3.43s of CPU for the same work; W's wall ≈ CPU/1 (dd is single-threaded, it can't use the second core), T's wall ≈ CPU/0.5, and T's throttled_usec ≈ its extra wall. wall = cpu_time / min(quota, threads) — the ceiling binds fully, nothing hides behind I/O.

So the round-3 framing is confirmed: the five earlier null results measured the cold-cache/read corner where the expensive work is D-state and free of quota; the hot-cache/write corner — the sibling-clone production regime — is fully exposed. Two boundaries the table also shows:

1. The latency metric did not move. claim→first-exec is 114-170ms on both shapes; the wake handshake's touch set is too small to feel even a 0.5-core ceiling. The exposure is in bulk working-set re-touch after wake, not in time-to-first-work.
2. The default shape is safe by construction. Guaranteed-at-N gives quota = N cores; a guest thread re-touching pages needs ≤1 core, so any VM at the default never sees this. The 2x lives only in oversold shapes (quota < 1 core per active thread) — which is those shapes doing exactly what they were configured to do.

Implication for the P2/P3 split as I read it: P2 (staged quota) would only shelter the bulk re-touch of oversold VMs, i.e. exempt precisely the work the operator chose to cap — that is a policy question, not a mechanism gap, and the mechanism layer (this repo) already gives operators the knobs. P3's doc note gets the sharper wording the prediction earns: cold mmap first-touch is I/O-bound and immune to the quota; hot mmap (sibling clones) is CPU-bound and fully exposed; write-heavy oversold guests pay wall = cpu/quota on re-touch. Will leave P2 to the issue owner.

[CMGS]

P3 landed on master (36745c1): the vm.md note now keys on cache state — cold mmap first-touch is I/O-bound and invisible to the quota; hot-cache sibling clones pay pure-CPU CoW at wall = cpu_time / min(quota, busy threads); time-to-first-exec is flat either way. Guidance for oversold wake-sensitive shapes: --cpu-weight instead of a tight quota, or copy mode to prepay inside the exempt provisioning window.

P2 (staged quota) is rejected: there is no principled boundary for "working set touched" the mechanism layer could arm on, the sheltered work is exactly what the operator capped, and the burst credit the kernel offers (≤ quota per period) is orders of magnitude short of second-scale CoW. A runtime knob-update verb was also considered and dropped — VM-level CPU is immutable after launch, and a quota-only mutator would be an inconsistent half-lever.

With the regime identified, the exposure quantified, provisioning proven exempt, and the doc note landed, I believe this can close.