Connection between ESXi and NSX Controller is UNKNOWN and/or connection between ESXi and NSX Manager is DOWN.

Products

• VMware NSX

Issue/Introduction

• The following error may be observed
  

  Connection between host (<ESXi FQDN>, <UUID>) and NSX Controller is UNKNOWN. Response : [] Connection between host (<ESXi FQDN>, <UUID>) and NSX Manager is DOWN. Please restore connection before continuing. Response : Client has not responded to {2} consecutive heartbeats. Port {1234} between Host to NSX Manager must be open, Please check underlay physical firewalls and host hypervisor firewalls for troubleshooting.

• On the ESX host, the Manager and/or Controller connections may not be in an expected state
  

  As root user:

# nsxcli -c get managers

192.168.1.1 Standby (NSX-RPC)
192.168.1.2 Standby (NSX-RPC)
192.168.1.3 Standby (NSX-RPC) *

Expected status: all 3 Managers in a "Connected" state

# get controllers
 Controller IP    Port     SSL         Status       Is Physical Master   Session State  Controller FQDN           Failure Reason
  192.168.1.1     1235   enabled    disconnected          true               up               NA                       NA
  192.168.1.2     1235   enabled     not used             true               up               NA                       NA
  192.168.1.3     1235   enabled     not used             true               up               NA                       NA

Expected status: one controller all 3 Managers in a "Connected" state

• ESXi uses port 1234 to connect to the Manager and 1235 to connect to the Controller, checking the connection they are not in the expected Established state:
  

  # esxcli network ip connection list | grep 1234
tcp         0       0  192.168.1.100:<port>  192.168.1.1:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy
tcp         0       0  192.168.1.100:<port>  192.168.1.2:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy
tcp         0       0  192.168.1.100:<port>  192.168.1.3:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy

  
  # esxcli network ip connection list | grep 1235
tcp         0       0  192.168.1.100:<port>  192.168.1.1:1235   CLOSE_WAIT    2101584  newreno  nsx-proxy

• If the Manager connection is Up but Controller is down, any VMs connected to a segment will show their vDS port status as Blocked on the vSphere client

NOTE: In rare cases some of the above symptoms may be expected if a partial uninstall has been performed i.e. the ESX has been force removed from NSX UI with no uninstall performed on the ESX.

Environment

VMware NSX

Cause

NSX requires ports 1234 and 1235 to be open for connections from the ESXi hosts to the NSX Manager. 

Resolution

1. Check port connectivity to each of the NSX managers from the ESX over ports 1234 and 1235 using the netcat (nc) command:
   

   As root user on the ESXi host:
   
   # nc -zvv <NSX Manager IP> 1234
# nc -zvv <NSX Manager IP> 1235
   
   Expected result: port [tcp/*] succeeded!

2. If the port connectivity fails while running the above nc command, investigate further, typically the traffic is blocked by an external firewall.
3. If the port connectivity test is a success, it is possible an NSX service is in a bad state on the ESX host, these can be restarted as root user
   
   NOTE: The host must be placed in to Maintenance Mode before restarting the cfgagent and nestdb agents.

   # /etc/init.d/nsx-proxy restart
# /etc/init.d/nsx-opsagent restart
# /etc/init.d/nsx-cfgagent restart
# /etc/init.d/nsx-nestdb restart

4. If the issue persists, please open a case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.

Additional Information

Please refer to below KB explaining roles of each agent:
Troubleshooting NSX Host Agents