Skip to content

fix(hostagent): check PF interface before writing sriov_numvfs#62

Open
tsorya wants to merge 1 commit into
NVIDIA:public-mainfrom
tsorya:fix/check-pf-interface-before-vf-creation
Open

fix(hostagent): check PF interface before writing sriov_numvfs#62
tsorya wants to merge 1 commit into
NVIDIA:public-mainfrom
tsorya:fix/check-pf-interface-before-vf-creation

Conversation

@tsorya

@tsorya tsorya commented Jul 2, 2026

Copy link
Copy Markdown

Summary

  • Guard SetNumOfVFs with a PF interface check before writing sriov_numvfs
  • Prevents a catastrophic VF probe cascade (~46 VFs × 120s timeout each) when mlx5_core is in error state after firmware fatal error
  • The 30s reconcile loop naturally retries once the DPU firmware recovers

Problem

When the DPU firmware hits a fatal error during provisioning, mlx5_core on the host enters error state and tears down the netdev — but remains bound to the PCI device. The network manager's CreateP0VF step writes sriov_numvfs regardless, triggering the kernel to enumerate dozens of VFs that each block 120s in wait_fw_init, resulting in 90+ minutes of stuck kernel threads and failed provisioning.

Fix

Check that the PF has a net interface (proving the driver is healthy and firmware is responsive) before calling SetNumOfVFs. If the interface is missing, return an error early — the network manager retries every 30s and will succeed once the DPU finishes booting.

Test plan

  • Verify unit tests pass (go test ./internal/provisioning/hostagent/util/)
  • Deploy on a cluster and provision a DPU — confirm VFs are created normally when PF is healthy
  • Simulate firmware error scenario (or trigger BFB install) — confirm no VF cascade, provisioning recovers after DPU boot

Made with Cursor

When the mlx5_core driver enters error state (e.g., firmware fatal error
during DPU provisioning), it tears down the netdev but remains bound to
the PCI device. Writing sriov_numvfs in this state causes the kernel to
enumerate VFs that each block 120s in wait_fw_init, creating a cascade
of ~46 VFs × 120s = 90+ minutes of stuck kernel threads.

Guard SetNumOfVFs with a PF interface check: if PF net/ is missing, the
driver is unhealthy and we bail out. The 30s reconcile loop will retry
once the DPU firmware recovers and the interface reappears.

Signed-off-by: Igal Tsoiref <itsoiref@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant