ESXi 6.5, earlier versions might not enable all VFs of PCI SR-IOV device
search cancel

ESXi 6.5, earlier versions might not enable all VFs of PCI SR-IOV device

book

Article ID: 334849

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

On some platforms, ESXi might fail to enable all virtual functions (VFs) of a PCI SR-IOV device as requested by the device's physical function (PF) driver. This failure results in hardware resources being unused or wasted.

This issue occurs on platforms where the BIOS or UEFI firmware programs the PF's base address registers (BARs) so that gaps of unassigned physical address space are left within the parent PCI-PCI bridge's memory apertures. For example, non-prefetchable and prefetchable base or limit registers might not be assigned. When SR-IOV VFs are enabled and ESXi programs the VF BARs to assign them physical address space from the remaining gaps, it might not use the available space optimally. This poor usage results in a shortage of the number of VFs that can be enabled.

For example, if an SR-IOV NIC has 2 PFs with 64 VFs each, ESXi might not be capable of enabling all 128 VFs on the device.

Environment

VMware vSphere ESXi 6.5

Resolution

To resolve this issue, a boot option named pciBarAllocPolicy has been added in ESXi 6.5. The boot option has two settings:
  • first-fit (0x0)
  • smallest-fit (0x1)
By default, this boot option is set to first-fit (0x0)in ESXi 6.5.

Note: From ESXi 6.7 and beyond, the pciBARAllocPolicy to smallest-fit (0x1) is set by default. 

To work around this issue in ESXi 6.5, you must edit this boot option to smallest-fit (pciBarAllocPolicy=0x1). This change causes ESXi to assign the base address registers of a virtual function to physical addresses. It chooses the smallest gap within the parent bridge's memory aperture that is large enough to fit the base address register of the virtual function.