Packets are no longer reaching a VM using MAC learning after a vmotion of the VM

Products

• VMware NSX

Issue/Introduction

• The VM affected is connected with multiple vNICs with MAC learning on one or some of the vNIC's
• An example scenario is a Load Balancer solution in a high availability configuration consisting of two VMs in an active/standby state.
• There is a VIP/Floating IP and MAC address configured on the pair of HA nodes.
• MAC learning is enabled on the interface with the VIP/Floating IP and MAC address.
• The connectivity to the VIP IP is lost after the active node is migrated to any other host. The connectivity issue is resolved by migrating the active node back to the host it was last running on.
• When checking the Logical Switch MAC table from a host trying to communicate with the VIP IP, it is noticed that the table is still incorrectly pointing to the vtep IP of the host the active VM was previously running on.
• This can be confirmed using the commands and confirming the Outer/Remote IP the floating inner MAC address is pointing to. 

get logical-switch <UUID> mac-address-table (Edge Node)

get logical-switch <UUID> mac-table (Host/Transport Node)

• Log lines similar to the below are encountered on the vMotion source host in /var/run/log/vmkernel.log:

cpu30:36029995)vswitch: VSwitch_VmotionGetSize:78: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]vswitch vMotion runtime data size for port <PORT ID> = 30

cpu30:36029995)WARNING: vswitch: VSwitch_VmotionSave:247: [nsx@6876 comp="nsx-esx" subcomp="vswitch" errorCode="ESX6"]Unable to save vswitch state for port: <PORT ID>

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center 3.2.X

VMware NSX 4.1.X

Cause

• There is an issue which results in learnt MACs not being saved and restored on the destination host during a VMotion.
• The issue impacts the vNIC with learnt MAC addresses, provided it isn't the first connected vNIC on the destination host.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.4 and in NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

• vMotion the VM back to the host it was originally running on.
• Alternatively, generating packets from the VM which are sourced from the interface with the learnt MAC may update the LS MAC table and restore connectivity.