NSX-T 3.2.0/3.2.0.1 VMs permanently lose network after vMotion or newly created VMs cannot connect

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T Datacenter 3.2.0 and 3.2.0.1
• VM connectivity is lost after vMotion or a newly created VM has no network connectivity
• Other related failures maybe observed e.g. DFW updates not applied, empty Group membership on UI. 
• Connection to one of the Controllers is lost for some or all Transport Nodes, both ESXi and Edge TNs

Example output on ESXi host:
 

#nsxcli -c get controllers
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN
  192.168.10.1    1235   enabled      not used            false              null              NA
  192.168.10.2    1235   enabled    disconnected          true               down              NA
  192.168.10.3    1235   enabled      not used            false              null              NA 

• ESXi logs indicate the VM's port is blocked

/var/log/vmkernel
2022-04-17T09:35:58.331Z cpu4:4169881)World: vm 4169882: 6955: Starting world vmm0:NewVM of type 8
2022-04-17T09:36:15.190Z cpu23:4169881 opID=a366e85e)NetPort: 3026: blocking traffic on DV port ########-####-####-####-##########bb
/var/log/hostd
2022-04-17T09:36:15.194Z info hostd[2106252] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 3740 : The dvPort ########-####-####-####-##########bb was blocked in the vSphere Distributed Switch in ha-datacenter. It was in Unknown state before.

• On the ESXi host, the DVS port shows as blocked

#net-dvs | less
         port ########-####-####-####-##########bb:
                 com.vmware.common.port.alias = ########-####-####-####-##########bb , propType = CONFIG
                 com.vmware.common.port.connectid = 1650188158 , propType = CONFIG
                 com.vmware.common.port.block = true , propType = CONFIG 
                 com.vmware.common.port.backingType = nsx , propType = CONFIG
                 com.vmware.common.port.volatile.status = inUse linkUp blocked portID=67109092 Port blocked by admin propType = RUNTIME

• On the NSX Manager which is in a disconnected state above, var/log/cloudnet/nsx-ccp.log:
   

Failed to re-subscribe [tag:ccp] nsx$[null]. Listener is NOT SUBSCRIBED yet! lastProcessedTs:epoch: 13 sequence: 8249458, retry 20/20

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

This issue occurs due to a software issue on the Controller node which prevents an ESXi Transport Node from connecting. When an ESXi TN has no connection to the Controller, any new port connection either from a vmotion or new VM creation will be in a blocking state.

Resolution

This issue is resolved in NSX-T Data Center 3.2.1, available at Broadcom Downloads.

Workaround:
In the environment impacted by this issue, the following workaround can be applied.

• On the ESXi host where a VM has lost networking run

 #nsxcli -c get controllers
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN
  192.168.10.1    1235   enabled      not used            false              null              NA
  192.168.10.2    1235   enabled    disconnected          true               down            NA
  192.168.10.3    1235   enabled      not used            false              null              NA 

• Identify the IP address of the NSX-T Manager that is in a disconnected state.
• Reboot that NSX Manager.

Please upgrade at your earliest convenience for a permanent resolution.