Mapping PCI slot numbers to guest-visible PCI bus topology

Products

VMware vSphere ESXi

Issue/Introduction

Configuration files that describe VMware virtual machines (files with the .vmx extension) include an encoding of the persistent PCI slot that PCI or PCIe virtual devices such as NICs, HBAs, etc will be plugged into by the hypervisor when the virtual machine is powered on.

This article describes the internal algorithm used to map those persistent PCI slot numbers to the PCI bus topology visible within the guest OS. This description may aid in mapping multi-instance virtual devices (for example, multiple VMXNET3 vNICs) in the guest OS to the virtual device instance in the virtual machine's configuration.

Environment

VMware vSphere ESXi

Resolution

Terminology

PCI Bridge:
A device that bridges between two PCI buses.
Persistent PCI Slot:
An encoded number in the virtual machine configuration file used by the hypervisor to plug a virtual device into a specific slot in the virtual PCI bus topology.
Primary Bus:
PCI bus on the near side of the PCI bridge.
Secondary Bus:
PCI bus on the far side of the PCI bridge.
Bus:Dev:Func:
Logical PCI bus topology describing the location of the device. Most devices are single-function; in other words, Func is always 0. Legacy PCI devices have multiple devices on the same bus, while PCIe (being point-to-point) has one PCIe device on a bus.

Mapping algorithm

The mapping algorithm can be described using these steps:

Look up the persistent PCI slot number of a device in the virtual machine's configuration file:
```
# egrep "ethernet4.pciSlotNumber" /path/to/VM/VM.vmx
ethernet4.pciSlotNumber = "1216"
```
Encode the resulting PCI slot number in binary, using this bitmap:

FFF.BBBBB.DDDDD 001.00110.00000 [1216 = 0x4C0]
If BBBBB is 00000, the device is on the Primary PCI bus, at Bus:Dev.Func 0:DDDDD.0:

Example:
```
# egrep "pciBridge0.pciSlotNumber" /path/to/VM/VM.vmx
pciBridge0.pciSlotNumber = "17"
```
000.00000.10001 [17 = 0x11]
Bus:Dev.Func = 00h:11h.00h
If BBBBB is non-zero, the device is on a Secondary PCI bus. To look up the PCI Bridge device, use BBBBB - 1.
Using the example above for ethernet4 at 1216, BBBBB = 00110.

PCI Bridge Device = BBBBB - 1 = 5 (Virtual device name "pciBridge5")

Look up the persistent PCI slot number for the PCI Bridge:

# egrep "pciBridge5.pciSlotNumber" /path/to/VM/VM.vmx
pciBridge5.pciSlotNumber = "22"

Encode the resulting PCI slot number in binary (as shown in step 2):

FFF.BBBBB.DDDDD 000.00000.10110 [22 = 0x16]
The combination of FFF from step 2 and DDDDD from step 6 yields the Dev.Func of the PCI Bridge:

Bus:Dev.Func of PCI Bridge = 00h:16h:01h(Dev = 0x16 from step 6, Func = 0x1 from step 2])
If BBBBB for the bridge is non-zero, repeat the process from step 4 to locate the parent bridge.
The virtual device being looked up in step 1 can be found on this secondary bus at Dev.Func DDDDD.0 from step 2:

ethernet4 on secondary bus behind 00h:16h:01h at Dev.Func 00h:0h

Notes:

Virtual PCIe devices such as e1000e and VMXNET3 vNICs or PVSCSI vHBAs will be singleton devices attached to point-to-point secondary PCIe root ports as PCI Bridges, so their DDDDD encoding will usually be 0.
Virtual legacy PCI devices such as e1000 vNICs or lsilogic vHBAs will be multiple devices attached to a shared secondary PCI bus, so their DDDDD encoding will specify which device on the secondary PCI bus they are on.