Uplink with MTU less than DVS configured MTU can provoke VM packet loss in ESXi hosts with NSX and Mellanox NICs

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

In environments with Mellanox Driver and ESXi 7.0 packet losses could be experienced if the switch MTU is less than the N-VDS configured MTU (e.g. 1500 on the switch and 9000 on DVS) and if health check is not enabled in the transport zone. An alternative method to find out if such a situation happens if the ESXi host has Mellanox network adapters is provided here.

This issue was discovered in an environment with N-VDS.

Environment

• ESXi 7.0

• NSX-T 3.X

Cause

During a troubleshooting it has been noticed that in ESXi hosts not having Switch MTU compatible with DVS MTU this kind of output was appearing in the log  file  vmkernel.log  (grep for the nmlx5 string) running VMs

2024-11-14T14:38:15.563Z cpu86:2102132)<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_L2TableIndexRemove - (nmlx5_core_en_main.c:7744) Delete 0:50:56:XX:YY:ZZ from L2 table
2024-11-14T14:38:15.564Z cpu86:2102132)<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_L2TableIndexAdd - (nmlx5_core_en_main.c:7631) Add 0:50:56:XX:YY:ZZ to L2 table
2024-11-14T14:38:15.562Z cpu86:2102132)<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_L2TableIndexRemove - (nmlx5_core_en_main.c:7744) Delete 2:0:0:1:4:1 from L2 table
2024-11-14T14:38:15.563Z cpu86:2102132)<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_L2TableIndexAdd - (nmlx5_core_en_main.c:7631) Add 2:0:0:1:4:1 to L2 table
2024-11-14T14:38:15.562Z cpu86:2102132)<NMLX_INF> nmlx5_core: vmnic0: nmlx5_en_L2TableIndexRemove - (nmlx5_core_en_main.c:7744) Delete 0:50:56:ZZ:YY:WW from L2 table

This kind of log is not present in the ESXi host not having the problem. In the impacted ESXi  hosts the dropped RX packed  was increasing on some VMs interfaces (the DVS portID 12345XXXX is only an example, the switchport IDs can be retrieved with the net-stats -l command)

 

vsish -e cat /net/portsets/DvsPortset-0/ports/12345XXXX/stats                                                                             
packet stats {  
   pktsTx:43957611  
   pktsTxMulticast:15706  
   pktsTxBroadcast:155218  
   pktsRx:37279690  
   pktsRxMulticast:2376  
   pktsRxBroadcast:4169836  
   droppedTx:0  
   droppedRx:277285                       ,<----------------------------------This number was constantly increasing 
}

Resolution

Ensure that the upstream switches have interface MTU compatible with the MTU configured on DVS.