Mellanox Technologies MT27500 Family [ConnectX-3 and ConnectX-3 Pro devices] adapter stops processing traffic but network adapter link state is still active

Products

• VMware vSphere ESXi

Issue/Introduction

• An ESXi host is experiencing full traffic loss
• All Virtual Machine traffic using a Mellanox adapter stops
• Mellanox adapter driver is in use nmlx4_en 3.15.11.6 or 3.16.11.6 or 3.17.13.1 or 3.19.16.7 or 3.19.16.8
• Traffic is not passing over a Mellanox adapter but the link status shows as active
• Both the vmkernel and VMs go unresponsive on the network
• Network Card MT27500 Family [ConnectX-3 and ConnectX-3 Pro Devices]

If Mellanox Technologies MT27500 Family [ConnectX-3] devices stop processing traffic when RDMA and Ethernet (TCP) traffic is run in parallel,see: MT27500 Family devices stop processing traffic when RDMA & Ethernet traffic run in parallel

Environment

vSphere ESXi 6.x
vSphere ESXi 7.x

Cause

This is a driver related issue which can cause all network traffic to be lost when using this adapter and driver combination.

Resolution

This issue is resolved in later versions of the driver.

• nmlx4_en 3.16.11.10 (6.5 driver) 
• nmlx4_en 3.17.70.1 (6.7 driver)
• nmlx4_en 3.19.70.1  (7.0 driver)

Additional Information

Customer can flap the port from the upstream switch to resolve issue for a time.