Skip to content

fix(vm): keep the volume migration set atomic and finalized#2670

Closed
danilrwx wants to merge 21 commits into
mainfrom
fix/vm/atomic-volume-migration-set
Closed

fix(vm): keep the volume migration set atomic and finalized#2670
danilrwx wants to merge 21 commits into
mainfrom
fix/vm/atomic-volume-migration-set

Conversation

@danilrwx

@danilrwx danilrwx commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

A live migration of a VM with several local disks could be reported as successful yet leave the VM demanding a restart and its next migrations hanging. A migration could also get stuck outright and never complete. Node drains, firmware and workload updates on such VMs stall.

The cause: the controller built the set of volumes to migrate non-atomically, violating the rules KubeVirt enforces for volume migration. The fix makes it follow them:

KubeVirt rule When violated Fix
All local (ReadWriteOnce) disks migrate together, as one set. "Restart required" is silently latched on the VM; a disk is left unmigrated. The set is handed to KubeVirt only once every local disk has joined the round.
The target set must not change mid-migration: only continue to the same targets or fully revert to the source. KubeVirt rejects the update and the migration hangs. A new round waits until the in-flight one finalizes.
The volume-update strategy must be cleared by the VM owner; KubeVirt never clears it. Any later volume change is treated as part of a migration. The controller clears it once the migration is done.
A migration wedged before start (e.g. its target volume was deleted) is retried forever; KubeVirt never fails it. It stays pending indefinitely and blocks node drains. The operation fails after 5 minutes and the VM reverts to its source volumes.

Why do we need it, and what problem does it solve?

Multi-disk VMs on local storage are common (a boot disk plus data disks). Without this fix their migration is flaky: it may succeed on paper but leave the VM needing a manual restart, or hang and block node drains indefinitely with no way to recover except manual intervention.

What is the expected result?

For a running VM with a root disk and several local (ReadWriteOnce) disks:

  1. Repeated migrations (node drain, firmware or workload update) all complete, with every disk migrated to its new PVC.
  2. The VM is never left in the "restart required" state after a migration, and subsequent migrations do not hang.
  3. A migration that cannot proceed fails within minutes with a clear reason, the VM reverts to its source volumes, and the next migration succeeds.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: vm
type: fix
summary: "Virtual machines with several local disks migrate reliably: a migration no longer reports success while leaving a disk unmigrated, no longer hangs on an inconsistent volume set, and if it cannot proceed, it fails within minutes and the virtual machine recovers automatically."

@danilrwx danilrwx changed the title fix(vm): atomic volume migration set fix(vm): keep the volume migration set atomic and finalized Jul 14, 2026
@danilrwx
danilrwx force-pushed the fix/vm/atomic-volume-migration-set branch from 9a55f66 to c3a42f6 Compare July 14, 2026 14:06
danilrwx added 3 commits July 14, 2026 17:33
@danilrwx
fix(vm): keep the volume migration set atomic and serialized
d2550a1 
Live volume migration of a multi-disk VM built the migration volume set
non-atomically, so KubeVirt could be handed an inconsistent set: some disks on
this round's target while others still point at a previous round's PVC, or a
local disk left out of the set entirely. KubeVirt then rejects the transition
("the volume can only be reverted to the previous version during the update")
and the migration hangs, or completes a migration that never copied a local
disk, leaving the domain broken on the target node — reported as success but
requiring a restart.

SyncVolumes now:
- waits until all ReadWriteOnce local disks migrate together, never patching a
  set that mixes disks from different rounds (completeness);
- refuses to start a new migration while a volume migration is already in
  progress (VMI condition VolumesChange=True) that targets different
  destinations, since KubeVirt only accepts continuing the current round or a
  clean revert to source (serialization).

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(e2e): migrate a multi-disk VM repeatedly and keep the volume set…
d50df70 
… consistent

Add an e2e case to RWOVirtualDiskMigration that migrates a VM with a root disk
plus three ReadWriteOnce local disks several times in a row, asserting each
migration succeeds, all disks stay Ready on their targets, and the VM is never
left restart-required — the symptom of a non-atomic volume set.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
docs(vm): trim verbose volume-migration comments
5889710 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
danilrwx force-pushed the fix/vm/atomic-volume-migration-set branch from a7b41e6 to 5889710 Compare July 14, 2026 15:33
danilrwx added 3 commits July 14, 2026 17:51
@danilrwx
test(e2e): fail volume migration on VolumesUpdateError and revert ins…
0c2ab5c 
…tead of skipping

Drop the skip-on-known-failure for VolumesUpdateError and reverted VD
migrations so these specs surface the failure they are meant to guard.
The client-socket-closed KubeVirt flake stays skipped for now.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(e2e): migrate a multi-disk VM repeatedly under a pending restart
3c29f79 
The volume-update inconsistency ("the volume can only be reverted to the
previous version during the update") only reproduces when the whole RWO set
is migrated repeatedly while a restart is pending. Induce the restart before
the migration rounds so the spec exercises that path.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(e2e): request a restart mid-migration on a multi-disk VM
9229d20 
Migrate the whole RWO set once so the disks leave their base PVCs, start a
second migration, and request a restart while it is still in flight. The
restart reconcile must not push a conflicting volume update over the
unfinalized set. Combined with failing (not skipping) on VolumesUpdateError,
this guards the volume-consistency path; the race itself is exercised
aggressively by the manual repro.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
danilrwx marked this pull request as ready for review July 14, 2026 17:22
danilrwx added 4 commits July 14, 2026 20:47
@danilrwx
test(e2e): make the mid-migration restart spec storage-class portable
142cc54 
Dropping the fragile in-flight catch: on instant (replicated) storage the
migration finalizes before it can be observed mid-flight, which timed out. Just
request the restart right after the migration starts so it races the unfinalized
set on copy-based storage and lands right after on instant storage; both must
finalize cleanly.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vmop): time out migrations waiting for the VM to be ready
9a51192 
A volume migration whose target disks never synchronize (e.g. the target
PVCs stay Pending or a previous round left the state uncleaned) keeps the
VMOP Pending with reason WaitingForVirtualMachineToBeReadyToMigrate forever,
with no time bound. Bound the wait and fail the operation so the disk
migration revert path can recover the VM, and requeue while pending so the
deadline fires without further watch events.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): force-revert kvvm to source when a dead migration wedges it
6626c16 
When a volume migration cannot proceed and no operation is in progress, KVVM
may be left pointing at a dead migration target (e.g. the target PVC was
removed) while KVVMI stays on the source. kubevirt keeps requeuing on the
missing target and the kvvmiSynced barrier never lets us revert, so the VM
wedges forever. Force-revert KVVM to the source volumes in that case.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
test(vm): cover migration ready-to-migrate timeout and diverged-kvvm …
0966d7c 
…revert

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx danilrwx modified the milestones: v1.10.0, v1.11.0 Jul 15, 2026
danilrwx added 10 commits July 15, 2026 16:47
@danilrwx
fix(vm): log restart-required reflection at info with the real reason
91326f3 
Reflecting the KVVM RestartRequired condition logged a generic sentence at
error level with an empty err, polluting error logs and alerts for an expected
VM state while hiding why kubevirt asked for a restart. Log it at info and
attach the underlying KVVM condition reason and message.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): clear a stale migration strategy left on kvvm
2ca1a78 
A finished migration could leave updateVolumesStrategy=Migration on KVVM while
KVVM and KVVMI already agree on the volumes. KubeVirt then keeps treating the VM
as mid-migration. When no migration is in progress and only the strategy is
stale (volumes already match the desired set), clear it.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vmop): shorten ready-to-migrate timeout to 5m
ce91f82 
A healthy migration reaches ReadyToMigrate within about a minute, so 5 minutes
is a wide margin over the happy path while recovering a wedge sooner.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): log restart-required reflection only on transition
e2f1111 
The KVVM RestartRequired condition is latched by kubevirt until an actual
restart, so reflecting it logged the same line every reconcile. Log only when
the VM newly transitions into the awaiting-restart state.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
docs(vm): trim volume-migration recovery comments
4447702 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): reject any diverging volume set while a migration is in flight
da1bf98 
The in-flight destination check only verified that recorded migrated
volumes keep their destinations, so a set that migrates an extra volume
on top of the running round, keeps a volume at source mid-migration, or
drops an in-flight (hotplugged) volume slipped through and was rejected
by KubeVirt, leaving the controller retrying noisily until the round
finalized. Compare both directions: a patch is allowed mid-migration
only if it purely continues the current round.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): compare pull-policy normalized volumes in migration recovery…
f57401b 
… guards

The stale-strategy clearing and force-revert guards compared built
volumes (normalized with the VMI's containerdisk pull policies) against
the raw in-cluster KVVM, which never carries them. For a VM with a
containerdisk volume the equality never held, so a stale
updateVolumesStrategy was never cleared and the no-op check of the
force revert never suppressed a patch. Compare against the normalized
copy used by the kvvmi sync check instead.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): only force-revert a diverged kvvm for a stuck migration
553715c 
The recovery force-revert fired for any kvvm/kvvmi volume divergence with no
running migration, including a benign hotplug volume mid-attach, tearing the
attachment down (VMBDA stuck InProgress). Gate it on updateVolumesStrategy=
Migration so only a genuinely stuck migration is reverted; other divergence is
left to sync as before.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): do not force-revert kvvm while a no-vmop migration round is …
37d4764 
…active

A storage class change round runs without any VMOP: the migration itself
is driven by KubeVirt's workload-updater. In the window between patching
KVVM with the round's targets and KubeVirt syncing them to the KVVMI,
the recovery force-revert saw a diverged KVVM with no in-progress VMOP
and rolled the just-started round back to the source, churning the round
start. Skip the revert while the disks still demand a migration round.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
docs(vm): trim volume-migration guard comments
0f3dc0f 
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
fix(vm): guard volume patches with an optimistic lock on kvvm
5684d2b 
The volume patch replaces the whole volumes array from a read snapshot.
KubeVirt concurrently persists hotplugged (addvolume) volumes into the
same array, so a patch built from a stale read silently dropped the
just-persisted hotplug volume and the attachment never completed: with
local (WFFC) disks the node affinity legitimately evolves while PVCs
bind, each shift re-patched the volumes, and a VMBDA could stay
InProgress indefinitely. Assert the kvvm resourceVersion in the patch
so a concurrent change fails it and the next reconcile rebuilds from a
fresh read.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx danilrwx closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaroslavborbat yaroslavborbat Awaiting requested review from yaroslavborbat yaroslavborbat is a code owner
@Isteb4k Isteb4k Awaiting requested review from Isteb4k Isteb4k is a code owner

Assignees

@danilrwx danilrwx

Labels

None yet

Projects

None yet

Milestone

v1.11.0

Development

Successfully merging this pull request may close these issues.

1 participant

@danilrwx