Poor TCP performance might occur in Linux virtual machines with LRO enabled

Products

• VMware vSphere ESXi

Issue/Introduction

Large Receive Offload (LRO) functionality is enabled by default on VMXNET2 (Enhanced) and VMXNET3 devices. In some scenarios, Linux TCP/IP stack has a low performance when handling LRO-generated packets. As a result, having Large Receive Offload enabled for these devices in a virtual machine running a Linux guest operating system can cause poor TCP performance.

Environment

VMware ESXi 4.0.x Installable
VMware vSphere ESXi 6.0
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 6.5
VMware ESX 4.0.x
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware ESX 4.1.x

Resolution

Disabling LRO in the Guest Operating System
 

On virtual machines using VMXNET3 virtual network devices and running a guest OS with Linux kernel 2.6.24 or later, use the ethtool command to disable Large Receive Offload.

To disable Large Receive Offload using the ethtool command:

1. Run the ethtool command to disable LRO for each VMXNET3 virtual network interface:
   
   # ethtool -K ethX lro off
   
   For example:
   
   # ethtool -K eth0 lro off
   
   Note: The configuration change made with the ethtool command does not persist across reboots. To make the change persistent across reboots, add the ethtool command to a startup script such as rc.local.

For virtual machines running an earlier version of the Linux kernel or using VMXNET2 devices, disable Large Receive Offload by setting the module load time parameter for the vmxnet or vmxnet3 Linux driver to include disable_lro=1.

To set the module load time parameter:

Note: These steps may not work for RedHat Enterprise Linux/CentOS 5.x virtual machines when updating the /etc/modprobe.conf file.
 

1. Using a text editor, open the /etc/modules or /etc/modules.conf file used by the Linux distribution.
2. Locate the alias entry for the VMXNET or VMXNET3 driver:
   
   alias eth0 vmxnet
   
   or
   
   alias eth0 vmxnet3
    
3. Append an options entry to disable LRO on a new line under the alias.
   
   For example:
   
   alias eth0 vmxnet
options vmxnet disable_lro=1
   
   or
   
   alias eth0 vmxnet3
options vmxnet3 disable_lro=1
    
4. Save the configuration file.
5. Reboot the Linux virtual machine to apply the changes.

To disable LRO on these versions on Linux:
 

1. In a text editor, open the /etc/rc.local file and add the following two lines to the file for the VMXNET or VMXNET3 driver:
   
   rmmod vmxnet
modprobe vmxnet disable_lro=1
   
   or
   
   rmmod vmxnet3
modprobe vmxnet3 disable_lro=1
    
2. Save the configuration file.
3. Run both commands from the command line or reboot the virtual machine to apply the changes.

 

Disabling LRO on the Host

LRO can be disabled on ESXi/ESX hosts using this procedure:

1. Log in to the ESXi host or vCenter Server by using the vSphere Client.
2. Navigate to the host in the inventory tree, and on the Configuration tab click Advanced Settings under Software.
3. Select Net and scroll down until you reach parameters starting with Vmxnet.
4. Set the following LRO parameters from 1 to 0:
   
   Net.VmxnetSwLROSL
Net.Vmxnet3SwLRO
Net.Vmxnet3HwLRO
Net.Vmxnet2SwLRO
Net.Vmxnet2HwLRO
    
5. Reboot the ESXi/ESX host to apply the changes.

Additional Information

Using promiscuous mode in the Linux virtual machine leads to resolving the performance issue too because activating this mode leads to disabling LRO.

For more information, see Linux virtual machines using VMXNET3 driver experience network throughput degradation when connected to Traffic Shaping enabled Distributed vSwitch (2030927).