Mellanox Technologies MT27500 Family [ConnectX-3] devices stop processing traffic when RDMA and Ethernet (TCP) traffic is run in parallel

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• An ESXi host is experiencing full traffic loss
• All Virtual Machine traffic using a Mellanox adapter stops
• Traffic is not passing over a Mellanox adapter but the link status shows as active
• Both the vmkernel and VMs go unresponsive on the network.
• vmkping through vmk interface using CX3 uplink will fail.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0

Cause

This issue is observed only on rare conditions, when you are using Mellanox adapter driver nmlx4_en 3.19.16.3 and older. Additionally, you might see this issue only on Mellanox ConnectX-3 MT27500 Network Card. 

 Note: This issue is not seen with Mellanox ConnectX-3 Pro cards.

Resolution

This is a known issue.

Currently, there is no resolution.

Workaround:
To workaround this issue, Reboot the host.

Additional Information

Impact/Risks:
All network traffic can be lost when using this adapter