NSX Controller connectivity disconnected on ESXi transport node due to NSX Manager CCP handshake failure

Products

• VMware NSX

Issue/Introduction

Symptoms

• The ESXi transport node reports an NSX controller status of "disconnected" in the NSX UI or via nsxcli -c get controllers.
  
• Physical network connectivity (ports 1234 and 1235) between the ESXi host and the NSX Manager/Controller is confirmed healthy via nc tests.
• The nsx-nestdb service on the ESXi host is active
• Central Control Plane (CCP) logs on the NSX Manager report COMMUNICATION_ERROR or session timeouts specifically during the VersionMastershipHandshake process for the affected transport node
  
  
• NSX manager .22 node id is 647#####-####-####-####-ff#####df53 and NSX manager .24 node id is 6ce9eb3d-####-####-####-9851be54b365
  
  
• As per the ESXi host get controllers output, the controller master is NSX manager .22 with id 647#####-####-####-####-ff#####df53. However, nsx-ccp.log file reports controller is other node 6ce9eb3d-####-####-####-9851be54b365
  
  /var/log/cloudnet/nsx-ccp.log
  

  2026-08-06T08:59:39.756Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-3 HandshakeState 1533 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="com.vmware.nsx.handshake.HandshakeState"] [8#####7-####-####-####d######3, INIT, sId=0a####80-####-####-####-11#####3ee, nodeType=COMMON, false]: Moving to VERSION_CHECK_OK for TN 8#####7-####-####-####d######3
  2026-08-06T08:59:39.759Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-1 VersionMastershipServiceImpl 1533 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="handshake-server"] Close mastership for transport node 8#####7-####-####-####d######3, controller is other node 6ce9eb3d-####-####-####-9851be54b365
  2026-08-06T08:59:39.759Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-1 StubManagerImpl 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Endpoint 8#####7-####-####-####d######3 de-registered
  2026-08-06T08:59:39.759Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-1 HandshakeState 1533 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="com.vmware.nsx.handshake.HandshakeState"] [8#####7-####-####-####d######3, VERSION_CHECK_OK, sId=0a####80-####-####-####-11#####3ee, nodeType=COMMON, false]: Moving to INIT for TN 8#####7-####-####-####d######3
  2026-08-06T08:59:39.761Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 NettyConnection 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection closed received NettyConnection(NettyChannel(local=nsx-mgr02-ip:1235, remote=esxi-vmk0-ip:44725), active=false)
  2026-08-06T08:59:39.761Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 AbstractTransportProtocol 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] tcp:CCP-647#####-####-####-####-ff#####df53: Unregistering accepted NettyConnection(NettyChannel(local=nsx-mgr02-ip:1235, remote=esxi-vmk0-ip:44725), active=false) from its transport
  2026-08-06T08:59:39.761Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 EndpointNotifierImpl 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Endpoint notification dropped for [8#####7-####-####-####d######3], allow notification false
  2026-08-06T08:59:39.761Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-0 NsxRpcChannel 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] ServerChannel(vmware.nsx.handshake.VersionMastershipHandshake, 0a####80-####-####-####-11#####3ee).doClose(closeStream=false, status=Status(code=COMMUNICATION_ERROR, msg=null))
  2026-08-06T08:59:39.761Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 Resolver 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection NettyConnection(NettyChannel(local=nsx-mgr02-ip:1235, remote=esxi-vmk0-ip:44725), active=false) with endpoints [8#####7-####-####-####d######3] unregistered
  2026-08-06T08:59:39.762Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-0 NsxRpcChannel 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] ServerChannel(vmware.nsx.handshake.VersionMastershipHandshake, 0a####80-####-####-####-11#####3ee): Invoking close stream
  2026-08-06T08:59:39.762Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 ServiceResolverImpl 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Unregistering services [vmware.nsx.cli.ProxyConfigCliService, vmware.nsx.cli.MetricsCliService, vmware.nsx.messaging.MessagingClientService, vmware.nsx.rpc.RpcCliService, vmware.nsx.daemon.DaemonHealth] from connection NettyConnection(NettyChannel(local=nsx-mgr02-ip:1235, remote=esxi-vmk0-ip:44725), active=false)
  2026-08-06T08:59:39.762Z  INFO nsx-rpc:CCP-647#####-####-####-####-ff#####df53:user-executor-0 VersionMastershipServiceImpl 1533 - [nsx@6876 comp="nsx-controller" level="INFO" subcomp="handshake-server"] closeStream id 0a####80-####-####-####-11#####3ee status Status(code=COMMUNICATION_ERROR, msg=null)
  2026-08-06T08:59:39.762Z  INFO CCP-647#####-####-####-####-ff#####df53:worker-2 AbstractConnection 1533 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection NettyConnection(NettyChannel(local=nsx-mgr02-ip:1235, remote=esxi-vmk0-ip:44725), active=false) closed for the reason NETTY_CLOSE

Environment

VMware NSX

Cause

The NSX Manager Central Control Plane (CCP) service experiences a logical hang or internal session mismatch for specific transport nodes. This prevents the VersionMastershipHandshake from successfully initializing, causing the NSX Manager to reject the connection attempt from the host despite valid certificates and network reachability.

Resolution

Resolution

1. Identify the NSX Manager node currently acting as the master from the nsx-ccp.log file.
2. Reboot the problematic NSX Manager node to clear the hung CCP service state.
3. Once the manager node returns to an "UP" status, allow time for the CCP services to re-initialize and re-establish sessions with the ESXi transport nodes.
4. Verify the controller connectivity status on the ESXi host by running the following command: nsxcli -c get controllers
5. If the status transitions to "up" or "connected," the issue is resolved.

Additional Information

For further assistance, download the latest patch release at Download Broadcom Products and Software.
If the issue persists, contact Broadcom Support via Creating and Managing Broadcom Support Cases.