NSX Edge dropping traffic when running on an ESXi host with Mellanox network cards

Products

• VMware NSX

Issue/Introduction

When NSX edge is running on an ESXi host with Mellanox network cards, there may be some packet drops observed on the ESXi host's vmnics.

Using command /usr/lib/vmware/vm-support/bin/nicinfo.sh on the ESXi host, there are "Receive packets dropped" on the vmnic which also matches "ring full" counter. 

An example:

NIC statistics for vmnic2:
      Packets received: 15576594101
      Packets sent: 1427162519
      Bytes received: 5972134130612
      Bytes sent: 2698821449768
      Receive packets dropped: 17242488
      Transmit packets dropped: 0
      ....

NIC Private statistics:
      
      ....
      outOfBuffer: 17242488
      ....

The number of packet drops may be even higher when NSX edge is running in a host with mix-workloads (NSX edge on the same host as other workload VMs).

Any services running on the edge node, especially services for North-South traffic, may see network connection dropping or flapping, such as IPSec VPN tunnel and BGP connections. 

 

 

Environment

VMware NSX

VMware NSX-T Datacenter

Cause

This is caused by problematic Mellanox driver/firmware. 

Resolution

To resolve this issue, confirm Mellanox driver/firmware version is on Broadcom Compatibility Guide matching the ESXi version. A guide to determine hardware ID can be found here. 

In the event that the driver/firmware version matches the compatible version showed in Broadcom Compatibility Guide, it is advised to update the driver/firmware version to the newer version also listed in the compatibility list. 

If the condition does not improve, please contact Broadcom Support and upload the support bundles for the following:

• All three NSX manager nodes
• Any affected NSX edge nodes
• ESXi hosts where the edge nodes reside
• Reference to this KB