Why do host NICs sometimes change their ordering after an ESX installation?

Products

• VMware vSphere ESXi

Issue/Introduction

This article explains why NICs may change their ordering after an ESX 3.x installation. For ESX 4.x and all versions of ESXi, see ESXi/ESX host loses network connectivity after adding new NICs or an upgrade (2019871).

Environment

VMware ESX Server 3.5.x
VMware ESX Server 3.0.x

Resolution

For ESX 3.5 and earlier, VMware does not guarantee the order of your devices on an ESX/ESXi host, especially host NICs. It is possible to have one order during an installation and a different order after the installation is complete.

In PCI and BIOS maps, there are two ways to enumerate all of the PCI devices in a system:

• Depth first
• Breadth first

Note:

• In PCI numbering, devices are numbered slot:device.function.
• Assuming you have an onboard dual port NIC and an add-in dual port NIC, the onboard host NIC has addresses 03:00.0 and 03:00.1, and the add-in NIC has addresses 07:00.1 and 07:00.1.

PCI generic standards use depth first enumeration. Depth first goes in order of slot, then enumerates all PCI devices and functions in a single slot in order. Enumeration then goes to the next slot. If you enumerate hardware using depth first, here is how your host NICs map:

nic0=03:00.0
nic1:03:00.1
nic2:07:00.0
nic3:07:00.1

The other type of enumeration, breadth first, typically works well, but can run into problems when you have devices with multiple functions. Breadth first goes in order of slot, but only enumerates the first device and function in the slot. Enumeration then proceeds to the next slot. When all of the 0 functions have been enumerated, enumeration starts over with the first slot, but the 1 function.

If you enumerate breadth first, here is how your host NICs map:

nic0=03:00.0
nic1=07:00.0
nic2=03:00.1
nic3=07:00.1

Depth first enumeration runs into problems when using PCIe devices. PCIe bus numbering works differently, so it is possible to run out of IRQ allocation using depth first enumeration. When using PCIe devices with depth first enumeration, it is possible to run out of resource allocation on the first PCIe slot. For this reason, ESX uses breadth first enumeration. This allows for more than 2 PCIe devices to be present in a system without running out of resource allocation. This is why host NIC order changes may occur between the installation and boot-up of an ESX host.

ESX/ESXi 4.0 and later no longer uses depth first and breadth first for PCI device enumeration.