VM NIC disconnect resulting from vMotion from ESXi TN to ESXi TN

Products

• VMware NSX

Issue/Introduction

Symptoms:
VMs on the NSX-T prepared Transport Node will have no connectivity after SPF is not found error is observed.  This is only seen when Service Insertion is deployed.  Some partner solutions combine Service Insertion (SI) and Guest Introspection (GI) together.

Environment

VMware NSX-T Data Center

Cause

NSX-T owned Service Plane Forwarding port (SPF port) is deleted by vCenter when SI and / or GI is deployed

Affected ESXi versions 7.0 and above including U1, U2

Case: It happens with  ESXi 7.0 and above with use of VDS7.0 in the NSX-T environment. When Service Insertion and / or Guest Introspection is deployed and not enabled within 24-hours, the SPF port would be deleted from the ESXi host.  This will occur if these conditions are met. 
 
As a part of the  vCenter and ESX dvport sync workflow enacted every 24 hours, the dvports owned by NSX-T SPF port are unexpectedly removed from the NSX-T prepared Transport Node.

The ESXi host's root shell the command output of 'net-dvs -l' would not show SPF port present after 24 hours. 


When SI/GI feature is enabled, the SPF port is created on the host. We can verify from nsxaVim.log, the SPF port was created:

2021-03-29T18:58:14Z nsxaVim: [2152094]: INFO Creating spf port
[uuid=spfPort503cf18f6cffc28f|switchUuid=50 3c f1 8f 6c ## ## ##-## ## ## 14 1e 8e c6
ed|spf=true|extraConfigs=com.vmware.port.extraConfig.vnic.external.id:spfVif1617044555]
2021-03-29T18:58:14Z nsxaVim: [2152094]: INFO Port operation add called with
uuid=spfPort503cf18f6cffc28f|switchUuid=50 3c f1 8f 6c ## ## ##-## ## ## 14 1e 8e c6
ed|spf=true|extraConfigs=com.vmware.port.extraConfig.vnic.external.id:spfVif1617044555
2021-03-29T18:58:14Z nsxaVim: [2152094]: INFO Sucessfully add dvport
[spfPort503cf18f6cffc28f]

The first time SPF tried to connect the port:

vmkernel.log:2021-04-01T00:30:10.289Z cpu20:3079963)WARNING: spf:
SPFPort_Connect:145: [nsx@6876 comp="nsx-esx"]Could not connect SPF port : Not found

2021-04-01T00:30:10.289Z cpu20:3079963)WARNING: spf: SPFPort_Connect:145: [nsx@6876 comp="nsx-esx"]Could not connect SPF port : Not found

Resolution

This issue has been resolved in ESXi 7.0 Update 2C, build number 18426014.

Workaround:
1. Restart ops-agent service from ESX host root shell.  This recreates the spfPort on all host switches which participate in the NSX-T Overlay TZ.

/etc/init.d/nsx-opsagent restart

2. Reconnect the VM network from the vSphere client.

Additional Information

Impact/Risks:
Virtual machines can become disconnected and unable to communicate.