book
Article ID: 309346
calendar_today
Updated On:
Issue/Introduction
Symptoms:
Following the reconfiguration of PCIe slots (adding or removing cards), or possibly changing BIOS configuration parameters, you experience these symptoms:
- Virtual machines that have been configured to use SR-IOV virtual NICs cannot power on.
- After the power on fails, under the Virtual Machines tab, you see the error similar to:
Failed to power on virtual machine example_vm, pciPassthru:14: No SR-IOV virtual function (VF) is available for given physical function (PF) (0000:af:00.0). Check here for more details.
Environment
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
Cause
This issue occurs as a result of the fashion in which ESXi stores the max_vfs parameters. The parameters are stored in a comma separated driver module parameter called max_vfs. The values in the list are the number of number of VFs to create for each physical PCI device controller by this driver.
However, this parameter is associated with the driver, not the individual device. The devices are therefore implicitly implied by the s:b:d:f ordering of its PCI functions that are controlled by the associated driver.
For example, assume that a Quagmire2 NIC adapter (driver qedentv) is present in slot 2, and it presents two NIC physical functions at s:b:d:f addresses of 0000:1a:00.00 and 0000:af:00.1.
Assume that, before the slots are reconfigured, the max_vfs values are set up in this fashion:
nic slot sbdf MAX_VFS
=================================================
vmnic0 2.0 0000:af:00.0 4
vmnic1 2.1 0000:af:00.1 4
Note: This is just an example. Your system may be different.
Again, assume that the system is shut down, a second Quagmire2 NIC adapter is inserted into slot 3, and then the system is brought back up again. If by chance the s:b:d:f values for the new NIC end up at a lower numerical value than the NIC in slot 2, the pre-existing MAX_VFS parameters now apply to the NIC in slot 3 as shown below:
nic slot sbdf MAX_VFS
=================================================
vmnic0 2.0 0000:af:00.0
vmnic1 2.1 0000:af:00.1
vmnic4 3.0 0000:37:00.0 4
vmnic5 3.1 0000:37:00.1 4
The module parameters for driver qedentv did not change, but the way in which they are interpreted by ESXi is now errant. The parameters which previously applied to slot 2 now apply to slot 3. Thus, existing virtual machine will not see their SR-IOV virtual nics.