High network latency is observed on NSX logical networks when using the qfle3 driver versions prior to 1.0.79.0

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

This article describes a known issue with the qfle3 driver versions prior to 1.0.79.0 that can lead to high CPU usage on ESXi hosts and network latency over VXLAN backed NSX logical networks.

Symptoms:

• The ESXi host network adapter is using a version of the qfle3 driver prior to 1.0.79.0
• Network latency is observed between VMs communicating over VXLAN networks or VMs communicating with the external network through an NSX Edge gateway
• Network latency may be observed when running ping to test network connectivity
• Network latency may increase unexpectedly with increased load
• You may see high CPU usage for the NetWorld thread for a particular VM in net-stats or esxtop:
  
  "8687####": {"id": 8687####, "used": 100.00, "ready": 0.00, "cstp": 0.00, "name": "NetWorld-Dev-3355####-Tx",
• If Network IO Control is enabled, you may see high CPU usage for hclk packet scheduler thread in net-stats output:
  
  "667##": {"id": 667##, "used": 100.00, "ready": 0.00, "cstp": 0.00, "name": "hclk-####-vmnic4",
• The Tx packet rate does cannot justify the high CPU usage for the network threads

Environment

VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
VMware NSX for vSphere 6.3.x
VMware NSX Data Center for vSphere 6.4.x

Cause

This is a known issue described in the qfle3 driver release notes.

Resolution

The issue is resolved in qfle3 driver version 1.0.86.0 or later.

Note: Refer to your hardware vendor for the recommended firmware version to use with this driver.