After upgrading a virtual machine to hardware version 11, network dependent workloads experience performance degradation

Products

• VMware vSphere ESXi

Issue/Introduction

• You notice performance degradation for some client/servers workloads.
• Packets have up to a 0.5 second delay over the expected arrival in the application.
  
  This issue is observed under these conditions:
   
  • The guest operating system is Windows Server 2012, Windows 8 or later.
  • The virtual machine is on hardware version 11/ ESXi 6.0 compatibility.
  • The virtual NIC is vmxnet3 and the driver version is 1.6.6.0 and later.
  • The Receive Side Coalescing (RSC) feature is enabled globally and on the vmxnet3 adapter.
  • This issue is more prevalent when:
  • • Running Microsoft SQL/TDS based workloads
    • Using Jumbo Frames
    • The client and server are on two different hosts

Environment

VMware vSphere ESXi 6.0

Cause

Depending on different physical NIC and workload characteristics like chipset, coalescing settings and packet inter arrival rate, some packets offloaded by RSC might experience an additional delay. When aggregating a number of packets, ESXi will only keep the Push Flag (PSH flag) if it was set on the first packet to be coalesced. If the first packet does not have the PSH flag set but subsequent packets do, the final coalesced packet will not have it set and as a result, might not be delivered to the application immediately.

Resolution

Resolved in ESXi 6.0 Update 2.

 

Note: Applying ESXi 6.0 Update 2 may cause a Purple Diagnostic screen. For more information, see ESXi 6.0 Update 2 host fails with a purple diagnostic screen containing the error: Vmxnet3VMKDevRxWithLock and Vmxnet3VMKDevRx (2144968). This issue is resolved in ESXi 6.0 Patch Release ESXi600-201605001. 

 

If you are unable to upgrade, you can work around any performance degradation by disabling RSC on the affected guest operating system.

To disable RSC on the virtual machine, perform one of these options:

• Use the PowerShell command Disable-NetAdapterRsc.
  
  To disable for all interfaces:
  
  Disable-NetAdapterRsc *
  
  To disable for a specific interface:
  
  Disable-NetAdapterRsc -Name Ethernetx
   
• Disable RSC globally using the netsh int tcp set global rsc=disabled command.
  
  Note: This command requires an elevated command prompt.

To verify that RSC is disabled on the virtual machine:

• Use the PoweShell command Get-NetAdapterRsc. The IPv4OperationalState and IPv6OperationalState should both be set to FALSE.
• Use the netsh int tcp show global command. The Receive Segment Coalescing State should be set to disabled.

Note: This change is immediate and does not require rebooting the guest operating system.

 

Alternatively, you can disable LRO/RSC for all virtual machines on an ESXi host using:

 

esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0

esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0

 

Note: This will disable LRO for all virtual machines on the ESXi host. Virtual machines must be powered off and then powered on or migrated using vMotion to another hosts for the changes to take effect.