SR-IOV connected VMs uplink does not match expected values

Products

• VMware vSphere ESXi

Issue/Introduction

• Running virtual machines (VMs) configured with Single Root I/O Virtualization (SR-IOV).
• Esxtop and other commands ran on the ESXI show uplink mappings that do not match what is configured for SR-IOV. Specifically, esxtop may display uplinks not set within the SR-IOV config.
• For example the following scenario can occur.
  • VM connected to a DPG with VMNIC 6 as active uplink.
  • VM has an SR-IOV VNIC configured to VMNIC 7.
  • Checking the following commands:
    • esxcli network vm list
    • esxcli network vm port list -w <WorldID>
    • Output of the port list states "Team Uplink VMNIC6"
    • esxtop > (press) n > shows uplink for this VMs nic as VMNIC 6.
• Checking the .vmx file of the VM shows a value for pciPassthru<X>.pfId that does not seem to correspond to any known value or NIC.
  

Environment

• VMware ESXI
• Virtual Machines configured with SR-IOV (Virtual Functions / VFs)

Cause

1. Esxtop and standard virtual switch port views monitor traffic at the software vSwitch layer. Because SR-IOV Virtual Functions bypass the virtual switch to communicate directly with physical hardware, esxtop and vSwitch port lists cannot track hardware passthrough traffic paths. This is expected behavior and a known user experience limitation for SR-IOV interfaces.

2. PCI device mismatch: The physical function IDs (pfId) in the VM .vmx configuration file are formatted in base-10 (decimal), whereas ESXi CLI utilities (esxcli network nic list) display PCI addresses in base-16 (hexadecimal).

Resolution

This is a condition that may occur in a VMware vSphere environment. Standard virtual switch monitoring tools such as esxtop do not reflect direct hardware passthrough traffic for SR-IOV interfaces.  To check the mapping of the NIC to its SR-IOV function, navigate to the VM in vCenter and expand its settings to see the SR-IOV NIC assigment which is set and displayed on the VM.

To verify that SR-IOV Virtual Functions are correctly assigned to the intended physical NICs via CLI the below can be performed:

1. Log in to the ESXi host CLI via SSH.

2. Verify VF assignment per physical NIC:

   esxcli network vm list - To obtain the world ID of the VM you are checking.
   esxcli network sriovnic vf list -n vmnic7 - This will show all of the Virtual Functions mapped to this vmnic. The world ID of the VMs you have mapped to it via SR-IOV should show here.
   

   Confirm that the VM World ID is active on a VF under each physical NIC.

3. Inspect the PCI Physical Function ID (pfId) in the VM .vmx file:

   pciPassthru1.pfId = "0000:152:00.2" 

4. Convert the PCI segment/bus value from decimal (base 10) to hexadecimal (base 16):

   • Example: Decimal 152 converts to Hexadecimal 98 (152 = 152/16 = 9.5 or 9 + 8 remainder = 98 in HEX).

   • 0000:152:00.2 (Decimal) corresponds to PCI address 0000:98:00.2 (Hexadecimal).

5. Cross-reference the converted PCI address with physical NICs:

   esxcli network nic list
   
   [root@esxiexample:~] esxcli network nic list
   Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address        MTU   Description
   ------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -----------------------------------------
   vmnic7  0000:98:00.2  icen    Up            Up           10000  Full    aa:bb:cc:dd:ee:ff  9000  Intel(R) Ethernet Controller

   Confirm that 0000:98:00.2 maps to vmnic7 (for the earlier example).

6. Alternatively confirm direct device mapping using vim-cmd via ESXI CLI:

   vim-cmd vmsvc/getallvms
   vim-cmd vmsvc/device.getdevices <vmid>
   

   Inspect the sriovBacking > physicalFunctionBacking >  id parameter to confirm physical device assignment. The ID in use is the same as the esxcli network nic output.
   
   
   

Additional Information

Subscribe to this knowledge article to get updates on this issue.