In a Multi Uplink Environment, when Service Insertion is configured, VM loses the existing TCP and/or ICMP connectivity after the vMotion of Guest Virtual Machine.

Products

• VMware NSX

Issue/Introduction

To unblock the customer if a situation as above may arise.

Symptoms:

VM looses existing TCP and/or ICMP Connectivity after the vMotion of Guest Virtual Machine when the environment is configured as follows:

 1. The environment is configured with Service Insertion. This issue effects all the service insertion partners. 
 2. The ESXi host is configured to use multiple uplink in active/active mode.

** This issue doesn't impact the new flow. That is if you restart the Ping or TCP connection the traffic will not be impacted.

You see that the SVM and spfport are mapped to 2 different Physical nics

>nsxdp-cli vswitch instance list

  Client            PortID         DVPortID                            MAC             Uplink
  spf-spfPort            6710XXXX     spfPort503a35036fc9c725             ##:##:##:##:##:52    vmnicA
  Third-Party-SVM.eth1     6710XXXX    4deeec41-xxxx-yyyy-zzzz-1234abcd1234    00:50:56:xx:xx:xx    vmnicB<<<<<<<
  Guest VM         6710XXXX    97dfac372-xxxx-yyyy-zzzz-1234abcd1234    ##:##:##:##:##:48      vmnicA

 

Environment

VMware NSX-T Data Center

Cause

The spfport has the mac-learning enabled and it learns the MAC of the Guest VM from the service VNI on the source host. When the Guest VM is vmotioned, the MAC of Guest VM is not removed from the source host on the spfPort. The existing TCP/ICMP flows from the Guest VM return to the source host SVM for processing during which the MAC needs to be learnt on the physical NIC. This needs to occur since the Guest VM uses the new spfport on the destination host.
The packet destined towards the SVM will be received on the uplink that is mapped to it. Since the spfport on the source host has this MAC learned, and the packet is not received on the uplink mapped to the Spfport, it will be treated as a reflected packet and a MAC move is not initiated. The MAC entry of the Guest VM from the service VNI is still learnt on the spfPort of the source host. This packet is not dropped and is sent to SVM for processing. SVM process the packet and hands it to spfport as the mac of GVM is still learnt on the SPFport. SPFport tries to find Guest VM within the ESXi host and drops the packet.

Resolution

The fix for this is included in NSX-T 3.1.3.6 EP and this issue is resolved in NSX-T version 3.2.1

Workaround:
This issue is never seen when the spfPort and the SVM are mapped to the same uplink. Considering this, the below workarounds are applicable:

 1. This issue will not been seen if the customer uses a single uplink. In this scenario, both the SPF and SVM port would be mapped to the same uplink. After vmotion, if the flow is still on and goes to the old SVM, the MAC will be learned on the uplink and removed on the spfPort. Therefore, the first workaround would be just to use a single uplink.    

OR
 
 2. If the customer wants to leverage the use of multiple uplinks, they could consider using LAG (Link aggregation). In this scenario, multiple uplinks would be mapped to the same LAG. vswitch would treat the multiple uplinks to be part of the same logical uplink. Again, the SVM and spfPort would be mapped to the same uplink mitigating the issue.  

OR
 
 3. Customer can use multiple uplinks in active/standby mode.

Additional Information

Impact/Risks:
VM connectivity issues are seen in the infra.