Connection between host and NSX Controller is UNKNOWN due to connection between host and NSX Manager is DOWN.

Products

• VMware NSX

Issue/Introduction

• You are running into the below error:
  

  Connection between host (<fqdn of the esxi>, ######-a###-45e8-8##7-######256ee) and NSX Controller is UNKNOWN. Response : [] Connection between host (<fqdn of the esxi>, ######-a###-45e8-8##7-######256ee) and NSX Manager is DOWN. Please restore connection before continuing. Response : Client has not responded to {2} consecutive heartbeats. Port {1234} between Host to NSX Manager must be open, Please check underlay physical firewalls and host hypervisor firewalls for troubleshooting.

• You see from the following command that connectivity to all of the managers is showing as Standby instead of Connected
  

  # nsxcli -c get managers

• When running the below commands, you see connectivity between the host and managers showing in CLOSE_WAIT or TIME_WAIT or FIN_WAIT state instead of Established:
  

  # esxcli network ip connection list | grep 1234
  
  # esxcli network ip connection list | grep 1235

  
  

NOTE: This may be expected behavior if the UUID noted in the NSX Manager error message belongs to a host Transport Node that is unprepared for NSX without fully uninstalling NSX from the host, or if the connectivity between the host and NSX Manager is otherwise intentionally interrupted. If all host Transport Nodes are expected to be up and communicating normally with NSX, then proceed to the resolution.

Environment

VMware NSX
VMware NSX-T Datacenter

Resolution

1. Check port connectivity to each of the NSX managers from the host over ports 1234 and 1235 using the netcat (nc) command:
   

   # nc -zvv ###.###.### 1234
   
   # nc -zvv ###.###.### 1235
   

2. If the port connectivity fails while running the above nc command, then investigate the ports to see if there are any blocks on the firewall or in the networking datapath.
3. If the port connectivity connects normally with the nc command, then proceed to restart below agents on the problem ESXi/s using below commands, one after the other >> monitor for improvements and then proceed to restart the next service.
   
   NOTE: The host must be placed in to Maintenance Mode before restarting the cfgagent and nestdb agents.

   # /etc/init.d/nsx-proxy restart
   
   # /etc/init.d/nsx-opsagent restart
   
   # /etc/init.d/nsx-cfgagent restart
   
   # /etc/init.d/nsx-nestdb restart

4. If the issue persists after restarting the agents, please open a case with Broadcom Support.

Additional Information

Please refer to below KB explaining roles of each agent:
Troubleshooting NSX Host Agents