Network I/O performance regression after migrating from bnx2x driver to qfle3 driver

Products

• VMware vSphere ESXi

Issue/Introduction

When you upgrade from VMware ESXi 6.5 to 6.7, you experience the below symptoms:

• Qlogic E3 NIC driver migrates from bnx2x to qfle3 driver.
• You experience performance regression, i.e higher CPU utilization, lower throughput in some cases.

OR

 You see below PSOD on ESXi 6.7 host:

########################FastSlab_AllocWithTimeout@vmkernel##############################
########################vmk_PageSlabAlloc@vmkernel####################################
########################PktPageAlloc_AllocPages@vmkernel################################
########################vmk_PktAllocPage@vmkernel#####################################
########################qfle3_page_alloc_and_map@(qfle3)#################################
########################qfle3_alloc_rx_sge_mbuf@(qfle3)###################################
########################qfle3_rxeof@(qfle3)#############################################
########################qfle3_netpoll_rxcb@(qfle3)########################################

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

• This issue occurs due to the difference in data path mechanism: bnx2x supports queue-paring mode.
• while qfle3 does not. As the result, qfle3 consumes more MSI-X vectors in data path.
• which in turn causes higher system overhead. Because of the lack of queue-paring mode, networking Load balance is also unable to consolidate traffic in netqueue.

Resolution

To resolve this issue please upgrade qfle3 driver to 1.0.84.0 or later.

ESX6.5: VMware vSphere - Enterprise Plus 6.5

To workaround this issue on VMware ESXi 6.x:

1. Disable qfle3 driver and enable bnx2x driver by running the below commands:

esxcli system module set --enabled=true --module=bnx2x
esxcli system module set --enabled=false --module=qfle3

2. Reboot the ESXi host.

Note: VMware ESXi 6.7 image is shipped with both bnx2x and qfle3 drivers.