NSX-T 3.2.0/3.2.0.1 VMs permanently lose network after vMotion or newly created VMs has no network connectivity

Products

• VMware NSX-T Data Center

Issue/Introduction

• You are using NSX-T 3.2.0/3.2.0.1
• VM loses connectivity after vMotion or a newly created VM has no network connectivity
• Other related failures maybe observed e.g. DFW updates not applied, empty group membership on UI. 
• Connection to one of the Controllers is lost for some or all Transport Nodes, both ESXi and Edge TNs

Example output on ESXi host:
 

#nsxcli -c get controllers
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN
  192.#.#.#       1235   enabled      not used            false              null              NA
  192.#.#.#       1235   enabled    disconnected          true               down              NA
  192.#.#.#       1235   enabled      not used            false              null              NA 

• ESXi logs indicate the VM's port is blocked

/var/log/vmkernel
####-04-17T09:35:58.331Z cpu4:4169881)World: vm 4169882: 6955: Starting world vmm0:NewVM of type 8
####-04-17T09:36:15.190Z cpu23:4169881 opID=a366e85e)NetPort: 3026: blocking traffic on DV port ########-####-####-####-##########bb
/var/log/hostd
####-04-17T09:36:15.194Z info hostd[2106252] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 3740 : The dvPort ########-####-####-####-##########bb was blocked in the vSphere Distributed Switch in ha-datacenter. It was in Unknown state before.

• On the ESXi host, the DVS port shows as blocked

#net-dvs | less
         port ########-####-####-####-##########bb:
                 com.vmware.common.port.alias = ########-####-####-####-##########bb , propType = CONFIG
                 com.vmware.common.port.connectid = 1650188158 , propType = CONFIG
                 com.vmware.common.port.block = true , propType = CONFIG 
                 com.vmware.common.port.backingType = nsx , propType = CONFIG
                 com.vmware.common.port.volatile.status = inUse linkUp blocked portID=67109092 Port blocked by admin propType = RUNTIME

• On the NSX Manager which is in a disconnected state above, var/log/cloudnet/nsx-ccp.log:
   

Failed to re-subscribe [tag:ccp] nsx$[null]. Listener is NOT SUBSCRIBED yet! lastProcessedTs:epoch: 13 sequence: 8249458, retry 20/20

Environment

VMware NSX-T Data Center 3.2.x

Cause

This issue occurs due to a software issue on the Controller node which prevents an ESXi Transport Node from connecting. When an ESXi TN has no connection to the Controller, any new port connection either from a vmotion or new VM creation will be in a blocking state.

Resolution

This issue is resolved in VMware NSX 3.2.1, available at Broadcom downloads.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

• Run the below command on the ESXi host where the problem VM resides

 #nsxcli -c get controllers
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN
  192.#.#.#       1235   enabled      not used            false              null              NA
  192.#.#.#       1235   enabled    disconnected          true               down            NA
  192.#.#.#       1235   enabled      not used            false              null              NA 

• Identify the IP address of the NSX Manager that is in a disconnected state.
• Reboot that NSX Manager.

Please upgrade at your earliest convenience for a permanent resolution.