Configuring TSO
- Enabling or disabling hardware TSO on the host
By default, the host is configured to use hardware TSO if installed NICs support that feature.
To toggle hardware TSO on the ESX/ESXi host, use the vSphere Web Client or run a console command on the host:
- Determining if hardware TSO is enabled on the host
To determine if hardware TSO is enabled:
- Via SSH or console:
$
esxcli system settings advanced list -o /Net/UseHwTSO
- Verify that the
Int Value
property is equal to 1.
- Enabling TSO in a Linux guest operating system
To enable TSO in the guest operating system of a Linux virtual machine:
$
ethtool -K ethY tso on
Where Y in ethY
is the sequence number of the NIC in the virtual machine.
Configuring LRO
- Enabling hardware LRO for the VMXNET2 and VMXNET3 virtual machine adapters
Hardware LRO is active for VMXNET2 and VMXNET3 virtual machine adapters by default. Note: Rx checksum offload should be enabled in the virtual machine when using LRO.
Reboot the ESXi host for changes to take effect.
- Determining if hardware LRO is enabled on the host
To determine if hardware LRO is enabled for VMXNET2 or VMXNET3 adapters:
- In an ESX/ESXi Shell on the host, run the command for the
/Net/Vmxnet2HwLRO
or /Net/Vmxnet3HwLRO
parameters:
$
esxcli system settings advanced list -o /Net/Vmxnet2HwLRO
$
esxcli system settings advanced list -o /Net/Vmxnet3HwLRO
- In the command output, verify that the
Int Value
property is equal to 1.
- Enabling software LRO in the VMkernel for VMXNET2 and VMXNET3 adapters
To enable software LRO in the VMkernel for VMXNET2 and VMXNET3 adapters:
- In the vSphere Web Client, set the
Net.Vmxnet2SwLRO
and Net.Vmxnet3SwLRO
parameters to 1 to enable LRO, or to 0 to disable LRO.
- Via SSH or console, set the
/Net/Vmxnet2SwLRO
and /Net/Vmxnet3SwLRO
parameters to 1 to enable LRO, or to 0 to disable LRO.
- Reboot ESXi host
- Enabling or disabling LRO for all VMkernel adapters on a host
To enable or disable LRO for the default TCP/IP stack on the host:
- In the vSphere Web Client, on the Manage tab for the host, click Advanced System Settings and set
Net.TcpipDefLROEnabled
to 1 to enable LRO or to 0 to disable it.
- Via SSH or console, set the
/Net/TcipDefLROEnabled
to 1 to enable LRO or to 0 to disable it:
$
esxcli system settings advanced set -o /Net/TcpipDefLROEnabled -i 0
- Reboot ESXi host
- Determining if LRO is enabled for the VMkernel adapters on the host
To determine if LRO is enabled for all VMkernel adapters on the host:
$ esxcli system settings advanced list -o /Net/TcpipDefLROEnabled
- Enabling or disabling LRO in a Linux guest operating system
- To enable or disable LRO on a virtual machine that runs Linux kernel 2.6.24 and later and uses a VMXNET3 network interface:
- To enable LRO:
ethtool -K ethY lro on
- To disable LRO:
ethtool -K ethY lro off
Where Y in ethY
represents the sequence number of the NIC in the virtual machine.
-
- To permanently disable LRO on these versions of Linux:
- Open the
/etc/rc.local
file in a text editor and add these two lines to the file for VMXNET or VMXNET3 driver:
rmmod vmxnet
modprobe vmxnet disable_lro=1
Or
rmmod vmxnet3
modprobe vmxnet3 disable_lro=1
- Save the configuration file.
- Reboot the virtual machine for the changes to take effect.
Note: This is the appropriate way to disable LRO in RHEL 5.x.
- Enabling or disabling LRO in a Windows guest operating system
To enable or disable LRO in a Windows Server 2012 and later or Windows 8 and later virtual machine, see Manage TSO on a Windows Virtual Machine.
Note: On Windows, the LRO technology is also referred to as Receive Side Coalescing (RSC).
Segmentation and aggregation
- Situation 1: Virtual machine communicating with a virtual machine on another hosts
Location of TCP packet segmentation:
Tx TSO Status |
TCP Packet Segmentation Location |
Enabled |
Transmitting physical NIC |
Disabled |
Guest operating system |
Notes:
- If the physical NIC supports TSO and TSO is enabled, the NIC performs segmentation of application packets to TSO size. If the physical NIC does not support TSO, the VMkernel segments the application packets.
- If TSO is disabled, the guest operating system of the transmitting virtual machine segments application data into packets with MTU size.
Location of TCP packet aggregation:
Rx LRO Status |
TCP Packet Aggregation Location |
Enabled |
Receiving physical NIC if it supports LRO Receiving virtual machine NIC backend if the physical NIC does not support LRO |
Disabled |
TCP packets are not aggregation |
Notes:
- If the physical NIC on the receiving side supports hardware LRO, the NIC aggregates the arriving packets. If the NIC is not capable of LRO, the virtual machine NIC performs software aggregation.
- If LRO is disabled for the receiving virtual machine, but hardware LRO is active for the network adapter type (VMXNET2 or VMXNET3), the receiving physical NIC aggregates the packets. The VMkernel on the receiving side then segments the aggregated packets into smaller chunks with TSO size, and forwards the chunks to the target virtual machine.
- Situation 2: Virtual machine communicating with a virtual machine on the same host and virtual switch
Note: TSO/LRO processing for traffic between two virtual machines on the same virtual switch is done by the VMkernel. Because the traffic never leaves the hypervisor kernel, and because hypervisor networking is very efficient, performance does not suffer over physical TSO/LRO.
Location of TCP packet segmentation:
Tx TSO Status |
TCP Segmentation Location |
Enabled |
Transmitting virtual machine NIC sends TSO size packets |
Disabled |
Guest operating system |
Notes:
- If TSO is enabled for the transmitting virtual machine, the virtual machine NIC segments application data to packets that have TSO size.
- If TSO is disabled for the transmitting virtual machine, the guest operating system segments application data to packets that have MTU size.
Location of TCP packet aggregation:
Rx LRO Status |
TCP Packet Aggregation Location |
Enabled |
Receiving virtual machine NIC |
Disabled |
TCP packets are not aggregated |
Notes:
- If LRO is enabled for the receiving virtual machine, the virtual machine NIC aggregates application packets up to the configured buffer size, and sends the aggregated packets to the virtual machine.
- If LRO is disabled, packets that have MTU size are directly transferred to the target virtual machine. If large packets arrive, the VMkernel segments transferred packets to TSO size and forwards them to the virtual machine.
Additional Information
LRO on Windows is called Receive Segment Coalescing (RSC). It was introduced in Windows 8 / Windows Server 2012. LRO/RSC is supported for the Windows with VMXNET3 driver was supported from version vSphere 6.0 and Virtual Machine hardware version 11.