Poor network performance when using VMXNET3 adapter for routing in a Linux guest operating system

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When a Linux guest is configured to route IPv4 packets using the net.ipv4.ip_forwarding sysctl kernel value, you experience these symptoms:
 

• Performance is poor when using a VMXNET3 adapter, which is less than 100Mbps.
• For short periods, the guest stops forwarding packets and resume intermittently.
• Performance is poor when routing to virtual machines on the same ESX/ESXi host compared to virtual machines on other hosts, or physical nodes on the network.
• When switching to the E1000 virtual network adapter, performance is better.

Environment

VMware vSphere ESXi 6.0
VMware ESX 4.1.x
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Embedded

Cause

The VMXNET3 adapter supports an offloading feature called Large Receive Offloading (LRO), and in some Linux distributions, the VMXNET3 driver may have LRO enabled by default. The Linux kernel cannot handle LRO packets when performing packet forwarding and this offloading feature must not be used. This issue does not impact E1000 virtual adapters, because the E1000 driver does not support LRO.

Note: In ESX/ESXi releases prior to 4.1 Update 2, this issue were observed even when IP forwarding is disabled. All versions of ESXi, including ESXi 5.x require LRO disabled when IP forwarding is enabled.

Resolution

To resolve this issue, disable LRO for the VMXNET3 adapter, or use the E1000 virtual adapter instead.

For more information on disabling LRO in Linux, see

Poor TCP performance might occur in Linux virtual machines with LRO enabled - Poor TCP performance might occur in Linux virtual machines with LRO enabled

Understanding TCP Segmentation Offload (TSO) and Large Receive Offload (LRO) in a VMware environment

Additional Information

To determine if routing has been enabled using the sysctl value, run this command within the Linux guest:

# sysctl net.ipv4.ip_forward


If it is enabled, a value of 1 is returned. If it is disabled, a value of 0 is returned. For example:

net.ipv4.ip_forward = 0

Poor TCP performance might occur in Linux virtual machines with LRO enabled