ESXi 6.5, earlier versions might not enable all VFs of PCI SR-IOV device
Article ID: 334849
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
On some platforms, ESXi might fail to enable all virtual functions (VFs) of a PCI SR-IOV device as requested by the device's physical function (PF) driver. This failure results in hardware resources being unused or wasted.
This issue occurs on platforms where the BIOS or UEFI firmware programs the PF's base address registers (BARs) so that gaps of unassigned physical address space are left within the parent PCI-PCI bridge's memory apertures. For example, non-prefetchable and prefetchable base or limit registers might not be assigned. When SR-IOV VFs are enabled and ESXi programs the VF BARs to assign them physical address space from the remaining gaps, it might not use the available space optimally. This poor usage results in a shortage of the number of VFs that can be enabled.
For example, if an SR-IOV NIC has 2 PFs with 64 VFs each, ESXi might not be capable of enabling all 128 VFs on the device.
This issue occurs on platforms where the BIOS or UEFI firmware programs the PF's base address registers (BARs) so that gaps of unassigned physical address space are left within the parent PCI-PCI bridge's memory apertures. For example, non-prefetchable and prefetchable base or limit registers might not be assigned. When SR-IOV VFs are enabled and ESXi programs the VF BARs to assign them physical address space from the remaining gaps, it might not use the available space optimally. This poor usage results in a shortage of the number of VFs that can be enabled.
For example, if an SR-IOV NIC has 2 PFs with 64 VFs each, ESXi might not be capable of enabling all 128 VFs on the device.
Environment
VMware vSphere ESXi 6.5
Resolution
To resolve this issue, a boot option named pciBarAllocPolicy has been added in ESXi 6.5. The boot option has two settings:
Note: From ESXi 6.7 and beyond, the pciBARAllocPolicy to smallest-fit (0x1) is set by default.
To work around this issue in ESXi 6.5, you must edit this boot option to smallest-fit (pciBarAllocPolicy=0x1). This change causes ESXi to assign the base address registers of a virtual function to physical addresses. It chooses the smallest gap within the parent bridge's memory aperture that is large enough to fit the base address register of the virtual function.
- first-fit (0x0)
- smallest-fit (0x1)
Note: From ESXi 6.7 and beyond, the pciBARAllocPolicy to smallest-fit (0x1) is set by default.
To work around this issue in ESXi 6.5, you must edit this boot option to smallest-fit (pciBarAllocPolicy=0x1). This change causes ESXi to assign the base address registers of a virtual function to physical addresses. It chooses the smallest gap within the parent bridge's memory aperture that is large enough to fit the base address register of the virtual function.
Feedback
Yes
No
Powered by
