Linux virtual machines using VMXNET3 driver experience network throughput degradation when connected to Traffic Shaping enabled Distributed vSwitch

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When enabling Traffic Shaping on a Distributed vSwitch (DVS), Linux virtual machines using the VMXNET3 driver experience network throughput degradation.

This issue is observed only under these conditions:

• vSphere ESXi 4.1
• The Linux virtual machine is configured with a vNIC with the VMXNET3 driver
• The virtual machine is connected to a Traffic Shaping enabled DVS dvportgroup

Note: This issue does not apply to releases other than vSphere 4.1.

Environment

VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware vCenter Server 4.1.x

Cause

Traffic Shaper packet handling in vSphere 4.1 has compatibility issues with the Linux OS Large Receive Offload (LRO) / TCP Sending Offload (TSO) functionality.

Note: LRO functionality is only supported on the VMXNET3 driver.

Resolution

This issue is resolved in following ESX/ESXi 4.1 patches.

• VMware ESX 4.1 Patch ESX410-201211401-SG: Updates VMkernel, CIM, Tools, and others (2036261)

• VMware ESXi 4.1 Patch ESXi410-201211401-SG: Updates Firmware (2036273)

To work around this issue, use one of these options:

• Use to a release that is not affected by the issue (vSphere 4.0 or vSphere 5.0 and above).
  
  
• Use the e1000 driver.
  
  
• Disable large segment offload functionality at the guest OS level.
  
  
  • To disable TSO, run this command:
    
    # ethtool -K device tso off
    
    
  • To disable LRO on kernels prior to 2.6.24, run these commands:
    
    # rmmod vmxnet3
    # modprobe vmxnet3 disable_lro=1
    
    
  • To disable TSO and LRO on kernels 2.6.24 and later, run this command:
    
    # ethtool -K device lro off

Note: The configuration changes made with these commands (ethtool, rmmod, and modprobe) do not persist across reboots. To make the changes persist across reboots, add the appropriate commands to a startup script such as rc.local.

Additional Information

• To verify the TSO/LRO configuration on kernels after 2.6.24, run this command:
  
  # ethtool -k device
  
  rx-checksumming: on
  tx-checksumming: on
  scatter-gather: on
  tcp segmentation offload: off
  udp fragmentation offload: off
  generic segmentation offload: off
  generic-receive-offload: off
  
  
• To verify the LRO configuration on kernels prior to 2.6.24, run this command:
  
  # dmesg | grep features
  
  If lro does not appear on the last line, then LRO functionality is currently disabled for that host.
  
  For example:
  
  # dmesg | grep features
  features: sg csum vlan jf tso tsoIPv6 lro highDMA
  features: sg csum vlan jf tso tsoIPv6 highDMA
  
  In this example, LRO is disabled.

Note: -k (Lowercase) and -K (Uppercase) provides different purpose when used with ethtool.

• ethtool -k Get protocol offload information
  
• ethtool -K Set protocol offload
  

Poor TCP performance might occur in Linux virtual machines with LRO enabled
VMware ESX 4.1 Patch ESX410-201211401-SG: Updates VMkernel, CIM, Tools, and others
VMware ESXi 4.1 Patch ESXi410-201211401-SG: Updates Firmware
VMXNET3 ドライバを使用した Linux 仮想マシンでトラフィック シェーピングが有効な分散 vSwitch への接続時にネットワーク スループットが低下する