An ESXi host can reassign and/or lose the SR-IOV configuration

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
Following the reconfiguration of PCIe slots (adding or removing cards), or possibly changing BIOS configuration parameters, you experience these symptoms:

• Virtual machines that have been configured to use SR-IOV virtual NICs cannot power on.
• After the power on fails, under the Virtual Machines tab, you see the error similar to: 
     
  Failed to power on virtual machine example_vm, pciPassthru:14: No SR-IOV virtual function (VF) is available for given physical function (PF) (0000:af:00.0). Check here for more details.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0

Cause

This issue occurs as a result of the fashion in which ESXi stores the max_vfs parameters. The parameters are stored in a comma separated driver module parameter called max_vfs. The values in the list are the number of number of VFs to create for each physical PCI device controller by this driver.

However, this parameter is associated with the driver, not the individual device. The devices are therefore implicitly implied by the s:b:d:f ordering of its PCI functions that are controlled by the associated driver.

For example, assume that a Quagmire2 NIC adapter (driver qedentv) is present in slot 2, and it presents two NIC physical functions at s:b:d:f addresses of 0000:1a:00.00 and 0000:af:00.1.

Assume that, before the slots are reconfigured, the max_vfs values are set up in this fashion:
nic        slot           sbdf            MAX_VFS
=================================================
vmnic0     2.0            0000:af:00.0       4
vmnic1     2.1            0000:af:00.1       4

Note: This is just an example. Your system may be different.

Again, assume that the system is shut down, a second Quagmire2 NIC adapter is inserted into slot 3, and then the system is brought back up again. If by chance the s:b:d:f values for the new NIC end up at a lower numerical value than the NIC in slot 2, the pre-existing MAX_VFS parameters now apply to the NIC in slot 3 as shown below:

nic        slot           sbdf            MAX_VFS
=================================================
vmnic0     2.0            0000:af:00.0
vmnic1     2.1            0000:af:00.1
vmnic4     3.0            0000:37:00.0       4
vmnic5     3.1            0000:37:00.1       4

The module parameters for driver qedentv did not change, but the way in which they are interpreted by ESXi is now errant. The parameters which previously applied to slot 2 now apply to slot 3. Thus, existing virtual machine will not see their SR-IOV virtual nics.

Resolution

To resolve this issue:

1. Enable SR-IOV on the card which lost its parameters. For more information, see Enable SR-IOV on a Host Physical Adapter.
2. Reassign the virtual machine functions to Virtual Machines. For more information, see Assign a Virtual Function as SR-IOV Passthrough Adapter to a Virtual Machine.