fix(hostagent): check PF interface before writing sriov_numvfs#62
Open
tsorya wants to merge 1 commit into
Open
fix(hostagent): check PF interface before writing sriov_numvfs#62tsorya wants to merge 1 commit into
tsorya wants to merge 1 commit into
Conversation
When the mlx5_core driver enters error state (e.g., firmware fatal error during DPU provisioning), it tears down the netdev but remains bound to the PCI device. Writing sriov_numvfs in this state causes the kernel to enumerate VFs that each block 120s in wait_fw_init, creating a cascade of ~46 VFs × 120s = 90+ minutes of stuck kernel threads. Guard SetNumOfVFs with a PF interface check: if PF net/ is missing, the driver is unhealthy and we bail out. The 30s reconcile loop will retry once the DPU firmware recovers and the interface reappears. Signed-off-by: Igal Tsoiref <itsoiref@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SetNumOfVFswith a PF interface check before writingsriov_numvfsmlx5_coreis in error state after firmware fatal errorProblem
When the DPU firmware hits a fatal error during provisioning,
mlx5_coreon the host enters error state and tears down the netdev — but remains bound to the PCI device. The network manager'sCreateP0VFstep writessriov_numvfsregardless, triggering the kernel to enumerate dozens of VFs that each block 120s inwait_fw_init, resulting in 90+ minutes of stuck kernel threads and failed provisioning.Fix
Check that the PF has a net interface (proving the driver is healthy and firmware is responsive) before calling
SetNumOfVFs. If the interface is missing, return an error early — the network manager retries every 30s and will succeed once the DPU finishes booting.Test plan
go test ./internal/provisioning/hostagent/util/)Made with Cursor