Connection between ESXi and NSX Controller is UNKNOWN and/or connection between ESXi and NSX Manager is DOWN.

Products

VMware NSX

Issue/Introduction

The following error may be observed:

Connection between host (<ESXi FQDN>, <UUID>) and NSX Controller is UNKNOWN. Response : [] Connection between host (<ESXi FQDN>, <UUID>) and NSX Manager is DOWN. Please restore connection before continuing. Response : Client has not responded to {2} consecutive heartbeats. Port {1234} between Host to NSX Manager must be open, Please check underlay physical firewalls and host hypervisor firewalls for troubleshooting.

On the ESX host, the Manager and/or Controller connections may not be in an expected state.

nsx-syslog.log reports the following error:

Connecting to ssl://xx.xx.xx.xxx:1234 sid:13050] Couldn't connect to 'ssl://xx.xx.xx.xxx:1234' (error: 110-Connection timed out)

As the root user:

# nsxcli -c get managers
192.#.#.# Standby (NSX-RPC)
192.#.#.# Standby (NSX-RPC)
192.#.#.# Standby (NSX-RPC) *

Expected status: All 3 Managers in a "Connected" state.

# nsxcli -c get controllers
 Controller IP    Port     SSL        Status             Is Physical Master   Session State  Controller FQDN          Failure Reason
  192.#.#.#     1235   enabled    disconnected          true               up               NA                       NA
  192.#.#.#     1235   enabled    not used              true               up               NA                       NA
  192.#.#.#     1235   enabled    not used              true               up               NA                       NA

Expected status: One controller and all 3 Managers in a "Connected" state.

ESXi uses port 1234 to connect to the Manager and 1235 to connect to the Controller. Checking the connection, they are not in the expected Established state:

# esxcli network ip connection list | grep 1234
tcp         0       0  192.#.#.#:<port>  192.#.#.#:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy
tcp         0       0  192.#.#.#:<port>  192.#.#.#:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy
tcp         0       0  192.#.#.#:<port>  192.#.#.#:1234   CLOSE_WAIT    2101584  newreno  nsx-proxy

# esxcli network ip connection list | grep 1235
tcp         0       0  192.#.#.#:<port>  192.#.#.#:1235   CLOSE_WAIT    2101584  newreno  nsx-proxy

If the Manager connection is Up but the Controller is down, any VMs connected to a segment will show their vDS port status as Blocked on the vSphere client.

NOTE: In rare cases, some of the above symptoms may be expected if a partial uninstall has been performed (i.e., the ESX has been force-removed from the NSX UI with no uninstall performed on the ESX). You may also see an "NSX Install Failed" error next to the hosts in the UI.

Environment

VMware NSX
VMware Cloud Foundation

Cause

NSX requires ports 1234 and 1235 to be open for connections from the ESXi hosts to the NSX Manager.

Resolution

Follow these steps sequentially to diagnose and resolve connectivity issues:

Test Port Connectivity: Verify connectivity to each NSX manager over ports 1234 and 1235 using the nc (netcat) command as root on the ESXi host:
```
# nc -zvv <Manager_IP> 1234
# nc -zvv <Manager_IP> 1235
```
Expected Result: Connection to <IP> 1234 port [tcp/*] succeeded!
Verify Service Status: If port connectivity succeeds, inspect the state of NSX services. As the root user on the ESXi host, run:
```
# nsxcli -c get managers
# nsxcli -c get controllers
```
Ensure all managers and the controller are in a "Connected" or "Up" state.
Restart NSX Services: If the connectivity test succeeds but services remain unresponsive, restart the NSX agents on the ESXi host.
Note: Place the host in Maintenance Mode before proceeding.
```
# /etc/init.d/nsx-proxy restart
# /etc/init.d/nsx-opsagent restart
# /etc/init.d/nsx-cfgagent restart
# /etc/init.d/nsx-nestdb restart
```
Engage Support: If the issue persists, open a case with Broadcom Support. Refer to Creating and managing Broadcom support cases.

Additional Information

For detailed roles of each agent, see Troubleshooting NSX Host Agents. For comprehensive connectivity troubleshooting between the Control Channel and the Transport Node, see KB 410581.