vNIC's are disconnected from VMware NSX segment after VM's fail to migrate

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

After VM's fail to migrate, for example, after vSphere HA failover fails due to insufficient resources, you might experience those symptoms.

• vNIC's of such VMs are disconnected after poweron, even though "Connect At PowerOn" is enabled.
• It may fail to manually connect the vNIC again.
• vNIC's connected to non-NSX portgroups are not impacted.
• You see log entries similar to below, in the vmware.log of the VM.

<date/time>| vcpu-0| I005: VMXNET3 user: failed to connect Ethernet1 to network xxxxxxx-1222-4d2a-bfa5-xxxxxxxxxxxx.
<date/time>| vcpu-0| I005: Msg_Post: Warning
<date/time>| vcpu-0| I005: [msg.device.startdisconnected] Virtual device 'Ethernet0' will start disconnected.
...
<date/time> In(05) vcpu-0 - [msg.ethernet.vlance.connectFailed] Failed to connect 'ethernet0'.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

Due to the current design of VMware NSX, external id may be cleared from the ports after VM migration failure.
vNICs are disconnected from such ports.

 

On the destination host, VMs are unregistered if migration fails.
NSX ports are detached on unregistering and the external id is cleared from ports. The result is written to the port files in the same storage as the VMs.
Later on, when VMs try to connect to the VMware NSX ports on the source host, it reads the port files, but it has been detached on another host.
As a result, such vNICs are disconnected.

 

Ports without external id are cleaned up by a periodic script on ESXi.
If the script finds the external id is cleared, it deletes the ports.
After the ports are deleted, it fails to manually connect the vNICs again.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

Manually connect the disconnected vNICs.

If it fails, connect the disconnected vNICs to another portgroup and then connect it back to the original VMware NSX segment.
It creates new VMware NSX ports and vNICs are attached successfully.

Additional Information

Impact/Risks:

VMs lose network communication.