Low throuput when copying data between two Windows VMs using SCP

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• The SCP traffic may be dropped on some Windows Transport Nodes when the uplink is a teaming interface.
• This typically occurs in NSX-T cross host scenarios.
• Another type of SCP drop issue involves packet drops on the ESXi host, leading to TCP retransmissions and eventual packet drops.
• SCP command drop issues can be observed in both cases (MTU related and DFW related drops). 

Environment

VMware NSX-T 3.2.0

Cause

The SCP drop could be related with two different issue and two solutions "MTU in the Windows VM" or "DFW".

Resolution

Workaround 1 ( Windows VM ):

For NSX-T cross host case, if the scp drop issue is observed on Windows VM host with uplink teaming interface. In this case, user should change the MTU to MTU derived by this formulae "current MTU - 64" (e.g. 1500-64=1436) using CMD below:

netsh interface ipv4 set subinterface "xxx" mtu=yyyy store=persistent.

 

 

Workaround 2 ( ESXi host ):

 

Another type of SCP dropping issue occurs when some packets are dropped on the ESXi host, resulting in TCP retransmissions and eventual packet drops. These drops are caused by the DFW module, which is expected behaviour.

The reason for the drops is that the DFW monitors the TCP windows based on the ACK from the peer. However, with a teaming interface, the ACK windows may be updated to the wrong flow. When the same TCP traffic is sent out on the correct flow, the TCP sequence number is out of the TCP window, causing the DFW to drop the packet, including retransmission packets, ultimately leading to the TCP connection being dropped.

In an NSX-T setup, TCP traffic sent from the VIF (using a teaming interface) in A/A mode will be dropped by the underlay DFW module. To address this, the user can add a stateless policy with specific rules to allow the necessary traffic to pass through the DFW module, or they can add the relevant teaming subinterface to the exclusion lists.

 

Additional Information

This issue is resolved in VMware NSX-T 3.2.4.0 available at Broadcom Downloads.
If you are having difficulty finding and downloading software, please review the KB Download Broadcom products and software.