Virtual MAC registration issue with NMLX5 NIC on VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Observing an issue with the Mellanox ConnectX-5 NIC on a VMware ESXi host configured with a high-availability setup. The setup involves two application VMs using SR-IOV VFs with a shared vMAC(secondary MAC) address configured for redundancy.

 

During normal operation, VM1 is the active instance and traffic flows correctly using the configured vMAC. However, after the switchover event where VM2 becomes active, we observe that the vMAC fails to get reprogrammed in the NIC.

Cause

The error message is a report from the Mellanox nmlx5 driver. The driver is stating that the eswitch built into the driver/card cannot find this MAC address in its hashtable. Since the VM is set to use SR-IOV, the VM is directly connected to the physical NIC with a virtual function (VF), which bypasses the ESXi network vSwitch for packet handling. 

vmkernel: cpuxxx:xxxxxxx)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_EswitchApplyUcAddr - (nmlx5_core_en_eswitch.c:1482) UC address xx:xx:xx:xx:xx:xx vport 2 is not found in the hashtable

vmkernel: cpuxxx:xxxxxxx)<NMLX_ERR> nmlx5_core: vmnic2: nmlx5_en_EswitchApplyUcAddr - (nmlx5_core_en_eswitch.c:1482) ** UC address xx:xx:xx:xx:xx:xx vport 2 is not found in the hashtable

The driver is stating that the eswitch built into the driver/card cannot find this MAC address in its hashtable.

Resolution

Work with the OS vendor of the VM, and/or your hardware vendor, to troubleshoot the reason why the directly connected VM to NIC configuration is not behaving as desired.
SR-IOV is a direct connection from the VM to the physical NIC. This traffic may need more than just the connection, such as VLAN or Security (such as MAC Changes/Forged Transmits etc.). Therefore, these settings can be added, but this does not mean that the vSwitch handles this traffic. It is still to be considered a direct VM to physical NIC connection.

Additional Information

SR-IOV Component Architecture and Interaction