Linux virtual machines using VMXNET3 driver experience network throughput degradation when connected to Traffic Shaping enabled Distributed vSwitch
search cancel

Linux virtual machines using VMXNET3 driver experience network throughput degradation when connected to Traffic Shaping enabled Distributed vSwitch

book

Article ID: 310503

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:
When enabling Traffic Shaping on a Distributed vSwitch (DVS), Linux virtual machines using the VMXNET3 driver experience network throughput degradation.

This issue is observed only under these conditions:
  • vSphere ESXi 4.1
  • The Linux virtual machine is configured with a vNIC with the VMXNET3 driver
  • The virtual machine is connected to a Traffic Shaping enabled DVS dvportgroup
Note: This issue does not apply to releases other than vSphere 4.1.


Environment

VMware ESX 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware vCenter Server 4.1.x

Cause

Traffic Shaper packet handling in vSphere 4.1 has compatibility issues with the Linux OS Large Receive Offload (LRO) / TCP Sending Offload (TSO) functionality.

Note: LRO functionality is only supported on the VMXNET3 driver.

Resolution

This issue is resolved in following ESX/ESXi 4.1 patches.


To work around this issue, use one of these options:
  • Use to a release that is not affected by the issue (vSphere 4.0 or vSphere 5.0 and above).

  • Use the e1000 driver.

  • Disable large segment offload functionality at the guest OS level.

    • To disable TSO, run this command:

      # ethtool -K device tso off

    • To disable LRO on kernels prior to 2.6.24, run these commands:

      # rmmod vmxnet3
      # modprobe vmxnet3 disable_lro=1

    • To disable TSO and LRO on kernels 2.6.24 and later, run this command:

      # ethtool -K device lro off
Note: The configuration changes made with these commands (ethtool, rmmod, and modprobe) do not persist across reboots. To make the changes persist across reboots, add the appropriate commands to a startup script such as rc.local.


Additional Information

  • To verify the TSO/LRO configuration on kernels after 2.6.24, run this command:

    # ethtool -k device

    rx-checksumming: on
    tx-checksumming: on
    scatter-gather: on
    tcp segmentation offload: off
    udp fragmentation offload: off
    generic segmentation offload: off
    generic-receive-offload: off


  • To verify the LRO configuration on kernels prior to 2.6.24, run this command:

    # dmesg | grep features

    If lro does not appear on the last line, then LRO functionality is currently disabled for that host.

    For example:

    # dmesg | grep features
    features: sg csum vlan jf tso tsoIPv6 lro highDMA
    features: sg csum vlan jf tso tsoIPv6 highDMA

    In this example, LRO is disabled.

Note: -k (Lowercase) and -K (Uppercase) provides different purpose when used with ethtool.

  • ethtool -k Get protocol offload information
  • ethtool -K Set protocol offload

Poor TCP performance might occur in Linux virtual machines with LRO enabled
VMware ESX 4.1 Patch ESX410-201211401-SG: Updates VMkernel, CIM, Tools, and others
VMware ESXi 4.1 Patch ESXi410-201211401-SG: Updates Firmware
VMXNET3 ドライバを使用した Linux 仮想マシンでトラフィック シェーピングが有効な分散 vSwitch への接続時にネットワーク スループットが低下する