Low network throughput is observed for TCP workloads when using NSX-T ENS mode and i40en_ens driver version 1.1.3

Products

• VMware vSphere ESXi

Issue/Introduction

• The physical NICs are Intel XXV710 or X710 running i40en_ens driver version 1.1.3
• TCP throughput when testing with iperf or when performing a file copy is much lower than expected
• Performance issues are seen with TCP based applications running in VMs attached to an ENS mode NSX-T segment
• The problem may not occur if the VMs are running on the same ESXi host
• Statistics from netstat -s run inside the VM shows InCsumErrors counter incrementing
• When viewing a packet capture from the receiving VM or the ESXi host running the receiving VM, Wireshark flags a checksum error for Rx packets
• The issue is more likely to occur when the MTU of the guest VM has been changed (for example, to 1400 or 2000)

Environment

VMware vSphere ESXi 7.0.x
VMware ESXi 6.7.x

Cause

Intel i40en_ens driver version 1.1.3 contains a bug that can cause an incorrect checksum to be computed for packets that are segmented by TCP Segmentation Offload (TSO).

Resolution

This issue is resolved in i40en_ens driver 1.2.7:

To workaround the issue if you cannot upgrade, disable TSO in the affected VMs.

For Linux VMs, you can disable TSO with the following command:

ethtool -K ethY tso off

To re-enable TSO once you have installed the updated driver:

ethtool -K ethY tso on

To disable TSO for Windows VM, refer to Manage TSO on a Windows Virtual Machine.