Skip to content

feat(vm): add GPU DRA resource claim support#2520

Closed
danilrwx wants to merge 39 commits into
mainfrom
feat/gpu/add-base-gpu-support
Closed

feat(vm): add GPU DRA resource claim support#2520
danilrwx wants to merge 39 commits into
mainfrom
feat/gpu/add-base-gpu-support

Conversation

@danilrwx

@danilrwx danilrwx commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Description

Users running ML/rendering workloads can now attach a physical GPU to a virtual machine. Previously the VirtualMachine API had no way to request a GPU at all.

A new spec.gpus field lets a user request a GPU by naming a GPUClass (provided by the GPU module). The virtualization controller generates a DRA ResourceClaimTemplate per entry that targets the DeviceClass created by that GPUClass, marks the request for exclusive VFIO passthrough, and renders the corresponding DRA claims and GPU devices into the KubeVirt VM.

Highlights:

  • VirtualMachine.spec.gpus[] with a single field gpuClassName, up to 16 entries; list order is not significant, and duplicate entries request several cards of the same class.
  • A GPU feature gate (alpha, locked off in CE).
  • Admission rejects GPU entries unless the GPU feature gate is enabled and the referenced GPUClass exists; the check runs only when GPUs are introduced or changed, so unrelated updates of an existing VM are never blocked.
  • A GPUClassReady VM condition reports readiness: GPUClassNotFound when a class is missing (or its CRD is absent), GPUClassNotReady while the class is still initializing, GPUClassReady otherwise — instead of a silently pending claim.
  • Changing spec.gpus on a running VM requires a restart to take effect (AwaitingRestartToApplyConfiguration); a VM with a GPU is non-migratable.
  • Allocation and passthrough-safety constraints are also enforced in KubeVirt as a safety layer (depends on fix(dra): support Deckhouse GPU attributes 3p-kubevirt#130).

Why do we need it, and what problem does it solve?

ML and rendering workloads inside VMs need a real GPU. Until now there was no way to pass one through the VirtualMachine API.

Selecting a GPU by GPUClass keeps VM manifests portable: the user asks for a class (for example nvidia-h100) and lets DRA and the scheduler place the VM on a node with a suitable exclusive, passthrough-capable card, instead of pinning the VM to a specific node, PCI address, or GPU UUID. The selection policy (which SKU, whole card vs MIG, sharing, exclusivity) lives in the GPUClass owned by the GPU module, so it is not duplicated in the virtualization module.

What is the expected result?

  1. Install a GPU DRA provider and create a GPUClass (for example nvidia-h100); the GPU module creates a DeviceClass with the same name.
  2. Enable the GPU feature gate in ModuleConfig virtualization.
  3. Create a VM with spec.gpus:
    spec:
      gpus:
        - gpuClassName: nvidia-h100
  4. The controller creates a ResourceClaimTemplate, and the VM schedules on a node with a matching exclusive physical GPU passed through via VFIO.
  5. If the GPUClass is missing or not yet ready, the VM reports GPUClassReady=False with a clear reason instead of hanging silently. Changing spec.gpus on a running VM applies only after a restart.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: core
type: feature
summary: "Attach a physical GPU to a virtual machine by naming a GPUClass in spec.gpus."

@danilrwx
danilrwx force-pushed the feat/gpu/add-base-gpu-support branch from 22e85a0 to 3925492 Compare June 23, 2026 14:43
@danilrwx
danilrwx force-pushed the feat/gpu/add-base-gpu-support branch from fa2b083 to 888c89c Compare June 26, 2026 15:45
@danilrwx
danilrwx force-pushed the feat/gpu/add-base-gpu-support branch 3 times, most recently from f47d820 to 06d5029 Compare July 9, 2026 11:01
@danilrwx danilrwx added this to the v1.10.0 milestone Jul 9, 2026
@danilrwx
danilrwx force-pushed the feat/gpu/add-base-gpu-support branch from 7e69790 to 287d9c7 Compare July 9, 2026 11:42
danilrwx added 21 commits July 13, 2026 08:51
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Drop the req- and -template suffixes from generated DRA names and align
the device request name with the resource claim name (gpu-<device>).
The template name becomes <vm>-<device>.

This raises the user-facing gpuDevices[].name MaxLength from 55 to 59:
the previous 55-char limit was dictated by the req-gpu- prefix (8 chars)
that left no headroom against the 63-char DNS label limit.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…m request name functions

Remove GPUResourceClaimRequestName (identical to GPUResourceClaimName)
and sortedGPUDevicesForCompare (duplicate of sortGPUDevices). Export
SortGPUDevices so vmchange can reuse it.�[0;38;2;204;204;204m

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The template name was built as "<vmName>-<deviceName>", which two VMs in the same namespace could collide on (e.g. VM "a" with device "b-c" and VM "a-b" with device "c" both yield "a-b-c"). The losing VM hit a permanent not-controlled-by reconcile error. Append a vmName hash suffix to make the name unique per VM.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
… list and unset

isEmpty was derived from "GPUDevices == nil" while the compared values are sorted slices that collapse both nil and []{} to nil. A non-nil empty list versus unset produced a ChangeRemove with ActionRestart despite no real change. Derive isEmpty from the sorted slice length.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The GPU devices validator no longer checks for an externally created DeviceClass; the module now ships the gpu.deckhouse.io DeviceClass itself (gated by the GPU feature gate and Kubernetes >= 1.34). The validator keeps only the feature gate check, and the now-unused deviceclasses RBAC grant is dropped.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…ceClaimTemplate

The GPU DRA driver binds a card to vfio-pci and exposes /dev/vfio/* only
when the claim carries an opaque VfioDeviceConfig for the gpu.deckhouse.io
driver. The CEL selector alone picks the device but does not request
passthrough, so add the opaque config to the generated template.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Replace the model-based GPU request with an explicit DeviceClass
reference. spec.gpuDevices[].model becomes deviceClassName, and the
generated ResourceClaimTemplate targets that DeviceClass directly
instead of selecting devices through a module-owned CEL selector.

The module no longer ships the gpu.deckhouse.io DeviceClass; the class
is provided out-of-band (manually for now, by the GPU module later).
Device selection (physical/exclusive/sharing constraints) now lives in
the DeviceClass, and KubeVirt keeps enforcing passthrough safety. The
opaque VfioDeviceConfig for the gpu.deckhouse.io driver is retained.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Two paired blockers surfaced when applying a DRA GPU to the KubeVirt VM:

- The embedded VirtualMachine CRD kept "deviceName" in gpus.items.required
  (only "name" is required in the VMI CRD since the 1.6.2 bump). A DRA GPU
  carries claimName/requestName without deviceName, so the apiserver rejected
  the KVVM update with "gpus[0].deviceName: Required value". Drop deviceName
  from required so it matches the embedded VMISpec.
- Add the GPUsWithDRA feature gate to the KubeVirt configuration; the DRA
  admitter rejects DRA GPUs unless it is enabled.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The test still references annotations.AnnVMRestorePowerState in the
restore-power-state cases, but the import was dropped, breaking test
compilation with "undefined: annotations".

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…te alignment

- compare_test.go used the old model field in its gpuDevices YAML; after
  the field was renamed to deviceClassName the YAML no longer parsed, so
  the "restart on change" case detected no change. Switch the fixtures to
  deviceClassName.
- Re-align the featuregate const block; the GPU entry left it aligned to
  two different columns, tripping the formatter.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
ValidateUpdate rejected any update to a VM that still has gpuDevices when
the GPU feature gate is off. A VM created while the gate was enabled would
then be unmanageable (even unrelated edits) if the gate is later disabled.
Skip the gate check when gpuDevices are unchanged, so the gate is enforced
only on introducing or changing GPU devices.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…me prefix

VM "vm-a" and "vm-a-b" collide on the IsGPUResourceClaimTemplateName
prefix check; ensure the IsControlledBy owner guard prevents vm-a from
deleting the ResourceClaimTemplate owned by vm-a-b.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx added 16 commits July 13, 2026 08:51
Without this check a VM referencing a non-existent DeviceClass passes
admission and its ResourceClaim hangs Pending forever with no clear
reason. Validate on admission that each gpuDevices[].deviceClassName
resolves to an existing DeviceClass, and fail fast otherwise. The check
runs together with the feature-gate check, i.e. only when GPU devices
are introduced or changed.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The GPU devices validator resolves gpuDevices[].deviceClassName through
the cached client, which needs get/list/watch on resource.k8s.io
deviceclasses. Without it admission fails for every VM referencing a GPU
DeviceClass. Add a read-only rule for deviceclasses.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
SetGPUDevices rebuilt the GPU part of the KVVM by deleting every
resourceClaim/GPU whose name starts with the "gpu-" prefix, which could
drop a foreign entry that happens to share the prefix. Narrow the match:
a resourceClaim must also reference a ResourceClaimTemplate named for this
VM, and a GPU must be a DRA entry (ClaimRequest set) rather than a
device-plugin GPU. Foreign or device-plugin entries are left intact.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
A VirtualMachinePool template carries a full VirtualMachineSpec, so
gpuDevices in the template were only validated when a replica VM was
created, turning a bad template into a delayed FailedCreate instead of an
up-front rejection. Add the GPU validator to the template-spec validator.
It runs in gate-only mode (nil client): the feature gate is enforced at
pool creation, while DeviceClass existence stays a per-replica check so a
pool can be defined before the GPU provider is installed.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…ions

The annotation-based rendering was replaced earlier, but a test title and
two empty-annotations assertions still referred to it.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The vmchange comparator sorts gpuDevices and treats reordering as no
change, while the webhook compared them order-sensitively and re-ran the
feature gate and DeviceClass checks on a pure reorder. Sort both sides
before comparing so a reorder cannot be rejected after the gate is
disabled or a DeviceClass is removed.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The gate description still said GPUs are requested by product model,
which was replaced by referencing a DRA DeviceClass.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Comparing the stored spec with the rendered one via DeepEqual breaks as
soon as the API server defaults any field absent from the rendered spec:
the comparison never converges and the handler deletes and recreates the
template on every reconcile. Stamp the rendered spec hash into an
annotation and compare hashes instead, following the tolerations-hash
pattern.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…nfig values

The VirtualMachinePool enum entry is not related to GPU support and does
not belong to this change set.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
… hash

Templates created before the hash annotation existed would otherwise be
recreated once on controller upgrade. Compare their stored spec directly
instead; they migrate to the hash on their next legitimate recreation.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The VM now references a GPUClass in spec.gpuDevices[].gpuClassName (renamed
from deviceClassName). The GPU module creates a DRA DeviceClass named
exactly after the GPUClass, so the generated ResourceClaimTemplate targets
that same-named DeviceClass, and the validator resolves GPUClass readiness
by checking that DeviceClass exists. CRD, docs and tests updated.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
golangci unparam: the helper always received "vm-current".

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Follow-up to the DeviceClass -> GPUClass rename in the VM spec.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The validator resolved a DeviceClass named after the GPUClass, so an
unrelated DeviceClass with that name would pass. Look up the GPUClass
custom resource (gpu.deckhouse.io/v1alpha1) directly. A missing CRD
(GPU module not installed) is reported distinctly from a missing
GPUClass. RBAC switches from deviceclasses to gpuclasses read access.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
A GPUClass deleted under a running VM previously left the next claim
silently pending. The GPU handler now sets a GPUClassReady condition:
False (reason GPUClassNotFound) when any referenced GPUClass is missing
or its CRD is absent, True otherwise, and removes it when no GPU devices
remain. The GPUClass GVK is centralized in kvbuilder and reused by the
validator; testutil registers it for fake clients.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
danilrwx force-pushed the feat/gpu/add-base-gpu-support branch from 4f27bb8 to cdf003f Compare July 13, 2026 06:52
…eady

Remove the per-device name field: gpuDevices entries now carry only
gpuClassName. Claim/GPU/template names are derived from the position in
the list sorted by GPUClass, so reordering the spec is a no-op and no
restart is triggered. The list becomes atomic (order not significant),
and duplicate gpuClassName entries request several cards of one class.

The GPUClassReady condition now also checks the GPUClass Ready status,
not just its existence, so a class that is still initializing (its
DeviceClasses not yet reconciled) reports GPUClassNotReady instead of a
silently pending claim.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
@danilrwx
danilrwx marked this pull request as ready for review July 13, 2026 09:59
z9r5
z9r5 previously approved these changes Jul 13, 2026
Shorter, matches the user-facing intent (a list of GPUs). Go field
VirtualMachineSpec.GPUDevices -> GPUs, JSON/CRD gpuDevices -> gpus;
GPUDeviceSpec type and Set/Sort helper names kept.

Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants