feat(vm): add GPU DRA resource claim support#2520
Closed
danilrwx wants to merge 39 commits into
Closed
Conversation
danilrwx
force-pushed
the
feat/gpu/add-base-gpu-support
branch
from
June 23, 2026 14:43
22e85a0 to
3925492
Compare
8 tasks
danilrwx
force-pushed
the
feat/gpu/add-base-gpu-support
branch
from
June 26, 2026 15:45
fa2b083 to
888c89c
Compare
danilrwx
force-pushed
the
feat/gpu/add-base-gpu-support
branch
3 times, most recently
from
July 9, 2026 11:01
f47d820 to
06d5029
Compare
danilrwx
force-pushed
the
feat/gpu/add-base-gpu-support
branch
from
July 9, 2026 11:42
7e69790 to
287d9c7
Compare
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Drop the req- and -template suffixes from generated DRA names and align the device request name with the resource claim name (gpu-<device>). The template name becomes <vm>-<device>. This raises the user-facing gpuDevices[].name MaxLength from 55 to 59: the previous 55-char limit was dictated by the req-gpu- prefix (8 chars) that left no headroom against the 63-char DNS label limit. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…m request name functions Remove GPUResourceClaimRequestName (identical to GPUResourceClaimName) and sortedGPUDevicesForCompare (duplicate of sortGPUDevices). Export SortGPUDevices so vmchange can reuse it.�[0;38;2;204;204;204m Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The template name was built as "<vmName>-<deviceName>", which two VMs in the same namespace could collide on (e.g. VM "a" with device "b-c" and VM "a-b" with device "c" both yield "a-b-c"). The losing VM hit a permanent not-controlled-by reconcile error. Append a vmName hash suffix to make the name unique per VM. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
… list and unset
isEmpty was derived from "GPUDevices == nil" while the compared values are sorted slices that collapse both nil and []{} to nil. A non-nil empty list versus unset produced a ChangeRemove with ActionRestart despite no real change. Derive isEmpty from the sorted slice length.
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The GPU devices validator no longer checks for an externally created DeviceClass; the module now ships the gpu.deckhouse.io DeviceClass itself (gated by the GPU feature gate and Kubernetes >= 1.34). The validator keeps only the feature gate check, and the now-unused deviceclasses RBAC grant is dropped. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…ceClaimTemplate The GPU DRA driver binds a card to vfio-pci and exposes /dev/vfio/* only when the claim carries an opaque VfioDeviceConfig for the gpu.deckhouse.io driver. The CEL selector alone picks the device but does not request passthrough, so add the opaque config to the generated template. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Replace the model-based GPU request with an explicit DeviceClass reference. spec.gpuDevices[].model becomes deviceClassName, and the generated ResourceClaimTemplate targets that DeviceClass directly instead of selecting devices through a module-owned CEL selector. The module no longer ships the gpu.deckhouse.io DeviceClass; the class is provided out-of-band (manually for now, by the GPU module later). Device selection (physical/exclusive/sharing constraints) now lives in the DeviceClass, and KubeVirt keeps enforcing passthrough safety. The opaque VfioDeviceConfig for the gpu.deckhouse.io driver is retained. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Two paired blockers surfaced when applying a DRA GPU to the KubeVirt VM: - The embedded VirtualMachine CRD kept "deviceName" in gpus.items.required (only "name" is required in the VMI CRD since the 1.6.2 bump). A DRA GPU carries claimName/requestName without deviceName, so the apiserver rejected the KVVM update with "gpus[0].deviceName: Required value". Drop deviceName from required so it matches the embedded VMISpec. - Add the GPUsWithDRA feature gate to the KubeVirt configuration; the DRA admitter rejects DRA GPUs unless it is enabled. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The test still references annotations.AnnVMRestorePowerState in the restore-power-state cases, but the import was dropped, breaking test compilation with "undefined: annotations". Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…te alignment - compare_test.go used the old model field in its gpuDevices YAML; after the field was renamed to deviceClassName the YAML no longer parsed, so the "restart on change" case detected no change. Switch the fixtures to deviceClassName. - Re-align the featuregate const block; the GPU entry left it aligned to two different columns, tripping the formatter. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
ValidateUpdate rejected any update to a VM that still has gpuDevices when the GPU feature gate is off. A VM created while the gate was enabled would then be unmanageable (even unrelated edits) if the gate is later disabled. Skip the gate check when gpuDevices are unchanged, so the gate is enforced only on introducing or changing GPU devices. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…me prefix VM "vm-a" and "vm-a-b" collide on the IsGPUResourceClaimTemplateName prefix check; ensure the IsControlledBy owner guard prevents vm-a from deleting the ResourceClaimTemplate owned by vm-a-b. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Without this check a VM referencing a non-existent DeviceClass passes admission and its ResourceClaim hangs Pending forever with no clear reason. Validate on admission that each gpuDevices[].deviceClassName resolves to an existing DeviceClass, and fail fast otherwise. The check runs together with the feature-gate check, i.e. only when GPU devices are introduced or changed. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The GPU devices validator resolves gpuDevices[].deviceClassName through the cached client, which needs get/list/watch on resource.k8s.io deviceclasses. Without it admission fails for every VM referencing a GPU DeviceClass. Add a read-only rule for deviceclasses. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
SetGPUDevices rebuilt the GPU part of the KVVM by deleting every resourceClaim/GPU whose name starts with the "gpu-" prefix, which could drop a foreign entry that happens to share the prefix. Narrow the match: a resourceClaim must also reference a ResourceClaimTemplate named for this VM, and a GPU must be a DRA entry (ClaimRequest set) rather than a device-plugin GPU. Foreign or device-plugin entries are left intact. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
A VirtualMachinePool template carries a full VirtualMachineSpec, so gpuDevices in the template were only validated when a replica VM was created, turning a bad template into a delayed FailedCreate instead of an up-front rejection. Add the GPU validator to the template-spec validator. It runs in gate-only mode (nil client): the feature gate is enforced at pool creation, while DeviceClass existence stays a per-replica check so a pool can be defined before the GPU provider is installed. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…ions The annotation-based rendering was replaced earlier, but a test title and two empty-annotations assertions still referred to it. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The vmchange comparator sorts gpuDevices and treats reordering as no change, while the webhook compared them order-sensitively and re-ran the feature gate and DeviceClass checks on a pure reorder. Sort both sides before comparing so a reorder cannot be rejected after the gate is disabled or a DeviceClass is removed. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The gate description still said GPUs are requested by product model, which was replaced by referencing a DRA DeviceClass. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Comparing the stored spec with the rendered one via DeepEqual breaks as soon as the API server defaults any field absent from the rendered spec: the comparison never converges and the handler deletes and recreates the template on every reconcile. Stamp the rendered spec hash into an annotation and compare hashes instead, following the tolerations-hash pattern. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
…nfig values The VirtualMachinePool enum entry is not related to GPU support and does not belong to this change set. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
… hash Templates created before the hash annotation existed would otherwise be recreated once on controller upgrade. Compare their stored spec directly instead; they migrate to the hash on their next legitimate recreation. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The VM now references a GPUClass in spec.gpuDevices[].gpuClassName (renamed from deviceClassName). The GPU module creates a DRA DeviceClass named exactly after the GPUClass, so the generated ResourceClaimTemplate targets that same-named DeviceClass, and the validator resolves GPUClass readiness by checking that DeviceClass exists. CRD, docs and tests updated. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
golangci unparam: the helper always received "vm-current". Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
Follow-up to the DeviceClass -> GPUClass rename in the VM spec. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
The validator resolved a DeviceClass named after the GPUClass, so an unrelated DeviceClass with that name would pass. Look up the GPUClass custom resource (gpu.deckhouse.io/v1alpha1) directly. A missing CRD (GPU module not installed) is reported distinctly from a missing GPUClass. RBAC switches from deviceclasses to gpuclasses read access. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
A GPUClass deleted under a running VM previously left the next claim silently pending. The GPU handler now sets a GPUClassReady condition: False (reason GPUClassNotFound) when any referenced GPUClass is missing or its CRD is absent, True otherwise, and removes it when no GPU devices remain. The GPUClass GVK is centralized in kvbuilder and reused by the validator; testutil registers it for fake clients. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx
force-pushed
the
feat/gpu/add-base-gpu-support
branch
from
July 13, 2026 06:52
4f27bb8 to
cdf003f
Compare
…eady Remove the per-device name field: gpuDevices entries now carry only gpuClassName. Claim/GPU/template names are derived from the position in the list sorted by GPUClass, so reordering the spec is a no-op and no restart is triggered. The list becomes atomic (order not significant), and duplicate gpuClassName entries request several cards of one class. The GPUClassReady condition now also checks the GPUClass Ready status, not just its existence, so a class that is still initializing (its DeviceClasses not yet reconciled) reports GPUClassNotReady instead of a silently pending claim. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
danilrwx
marked this pull request as ready for review
July 13, 2026 09:59
danilrwx
requested review from
Isteb4k,
fl64,
goganat,
nevermarine,
universal-itengineer,
yaroslavborbat and
z9r5
as code owners
July 13, 2026 09:59
z9r5
previously approved these changes
Jul 13, 2026
Shorter, matches the user-facing intent (a list of GPUs). Go field VirtualMachineSpec.GPUDevices -> GPUs, JSON/CRD gpuDevices -> gpus; GPUDeviceSpec type and Set/Sort helper names kept. Signed-off-by: Daniil Antoshin <daniil.antoshin@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Users running ML/rendering workloads can now attach a physical GPU to a virtual machine. Previously the
VirtualMachineAPI had no way to request a GPU at all.A new
spec.gpusfield lets a user request a GPU by naming aGPUClass(provided by the GPU module). The virtualization controller generates a DRAResourceClaimTemplateper entry that targets the DeviceClass created by that GPUClass, marks the request for exclusive VFIO passthrough, and renders the corresponding DRA claims and GPU devices into the KubeVirt VM.Highlights:
VirtualMachine.spec.gpus[]with a single fieldgpuClassName, up to 16 entries; list order is not significant, and duplicate entries request several cards of the same class.GPUfeature gate (alpha, locked off in CE).GPUfeature gate is enabled and the referenced GPUClass exists; the check runs only when GPUs are introduced or changed, so unrelated updates of an existing VM are never blocked.GPUClassReadyVM condition reports readiness:GPUClassNotFoundwhen a class is missing (or its CRD is absent),GPUClassNotReadywhile the class is still initializing,GPUClassReadyotherwise — instead of a silently pending claim.spec.gpuson a running VM requires a restart to take effect (AwaitingRestartToApplyConfiguration); a VM with a GPU is non-migratable.Why do we need it, and what problem does it solve?
ML and rendering workloads inside VMs need a real GPU. Until now there was no way to pass one through the
VirtualMachineAPI.Selecting a GPU by GPUClass keeps VM manifests portable: the user asks for a class (for example
nvidia-h100) and lets DRA and the scheduler place the VM on a node with a suitable exclusive, passthrough-capable card, instead of pinning the VM to a specific node, PCI address, or GPU UUID. The selection policy (which SKU, whole card vs MIG, sharing, exclusivity) lives in the GPUClass owned by the GPU module, so it is not duplicated in the virtualization module.What is the expected result?
GPUClass(for examplenvidia-h100); the GPU module creates a DeviceClass with the same name.GPUfeature gate inModuleConfig virtualization.spec.gpus:ResourceClaimTemplate, and the VM schedules on a node with a matching exclusive physical GPU passed through via VFIO.GPUClassReady=Falsewith a clear reason instead of hanging silently. Changingspec.gpuson a running VM applies only after a restart.Checklist
Changelog entries