Performance issue with transmits on host with Mellanox physical network adapters (ConnectX-4 and above)
search cancel

Performance issue with transmits on host with Mellanox physical network adapters (ConnectX-4 and above)

book

Article ID: 399002

calendar_today

Updated On: 06-30-2025

Products

VMware vSphere ESXi 8.0

Issue/Introduction

ESXi hosts with Mellanox NIC adapters can experience slower transmissions for packets with different VLAN (CoS PCP/801.2p) priorities.

NIC private statistics display traffic with multiple priorities when the Tx slowness is experienced.

Example: NIC private statistics

      rxPrio0_bytes: 2020106898153116

      rxPrio0_packets: 1612620898942

      txPrio0_bytes: 832701150526206

      txPrio0_packets: 908382940785

      rxPrio5_bytes: 4629068132

      rxPrio5_packets: 69922603

      txPrio5_bytes: 1638142

      txPrio5_packets: 11629

      rxPrio7_bytes: 527209257679

      rxPrio7_packets: 7322126123

      txPrio7_bytes: 60657544

      txPrio7_packets: 431321

Cause

This is a known issue in NIC firmware due to context switches over the hardware priority buffers that caused slower packet transmissions for traffic different VLAN (CoS PCP/801.2p) priorities.

Resolution

In ESXi 8.0.3 P05, a new inbox nmlx5 driver module parameter "trust_dscp_same_priority" was added to map all Differentiated Services Code Point (DSCP) priorities to the same traffic class to avoid slowness in the NIC hardware. 

This driver module parameter needs to be enabled with this command, followed by the reboot of the ESXi host:

esxcli system module parameters set -m nmlx5_core -p ‘trust_dscp_same_priority=1’

Note: This module parameter is also available in ESXi 7.0.3 async nmlx5 driver 4.22.73.1006. Please refer to async driver release notes from Nvidia for additional details.

Additional Information

As updated in resolution section reboot of ESXi host is required after setting the module parameter to take effect and this parameter should persist across ESXi reboots.