Upgrade of NSX on ESXi transport node fails and shows as "Host Disconnected" under NSX Configuration
Article ID: 410788
Updated On:
Products
Issue/Introduction
- The ESXi transport node's upgrade fails with error "Unexpected Error while upgrading.."
- The host shows as "Host Disconnected" under NSX Configuration
- The status of the controller on the host in disconnect state shows "Host_Rejected_Controller_Cert"
- nsx-syslog on that ESXi Transport Node logs below log traces -
2025-08-27T22:43:42.068Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] Write ccp session message to nestdb ccp_id { [#######] } ip { ipv4: [#######] } server_port: 1235 fqdn: "" state: DISCONNECTED master: true failure_reason: HOST_REJECTED_CONTROLLER_CERT2025-08-27T22:43:42.068Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] Write ccp session message to nestdb ccp_id { [#######] } ip { ipv4: [#######] } server_port: 1235 fqdn: "" state: DISCONNECTED master: false2025-08-27T22:43:42.068Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] Write ccp session message to nestdb ccp_id { [#######] } ip { ipv4: [#######] } server_port: 1235 fqdn: "" state: DISCONNECTED master: false2025-08-27T22:43:42.068Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] CcpConnection: Connecting to new CCP [#######].2025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] CcpConnection: Disconnecting from ssl://[#######]:12352025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" tid="7669786" level="INFO"] CcpConnection: Connecting to [#######]2025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="7669825" level="INFO"] ConnectionKeeper[7561] ssl://[#######]:1235] attempting connection2025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="INFO"] StreamSocket[64036] Init f:-1 i:-1 ? -> ssl://[MANAGER_IP]:1235] Created2025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="7669825" level="INFO"] RpcConnection[64036] Init to ssl://[#######]:1235 0] Queue threshold size 02025-08-27T22:43:42.069Z nsx-proxy[7669786] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="INFO"] StreamSocket[64036] Open f:47 i:0 ? -> ssl://[#######]:1235] async_connect2025-08-27T22:43:42.071Z nestdb-server[7669811] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="7669811" level="INFO"] Modify: TransactionID='45799' Client ID=nsx-proxy Size=0.00MB LogTx?=12025-08-27T22:43:42.076Z nestdb-server[7669811] NSX ####### - [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="7669811" level="INFO"] Notifying updates took 0 ms to 2 clients: [#######] [#######]2025-08-27T22:43:42.076Z nestdb-server[7669811] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-nestdb" tid="7669811" level="INFO"] Modify Complete: TransactionID='45799' Telemetry=[(3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4), (3143296.2, 30.4)]2025-08-27T22:43:42.078Z cfgAgent[7669468] NSX #######- [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="24ADEDC0" level="info"] Decoder: Received CCP_SESSION msg (Operation SET): ccp_id { left: [#######] right: [#######] } ip { ipv4: [#######] } server_port: 1235 fqdn: state: DISCONNECTED master: 1 failure_reason: HOST_REJECTED_CONTROLLER_CERT2025-08-27T22:43:42.078Z cfgAgent[7669468] NSX #######- [nsx@6876 comp="nsx-controller" subcomp="cfgAgent" tid="24ADEDC0" level="info"] Decoder: Received CCP_SESSION msg (Operation SET): ccp_id { left: [#######] right: [#######] } ip { ipv4: [#######] } server_port: 1235 fqdn: state: DISCONNECTED master: 0.
2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="INFO"] StreamSocket[64036] Open f:47 i:0 ? -> ssl://[#######]:1235] on_connect 336151574-sslv3 alert certificate unknown (SSL routines, ssl3_read_bytes)2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="WARNING"] StreamConnection[64036] Couldn't connect to 'ssl://[#######]:1235' (error: 336151574-sslv3 alert certificate unknown (SSL routines, ssl3_read_bytes))2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="7669825" level="WARNING"] RpcConnection[64036] Connecting to ssl://[#######]:1235 0] Couldn't connect to ssl://[#######]:1235 (error: 336151574-sslv3 alert certificate unknown (SSL routines, ssl3_read_bytes))
- syslog of the NSX manager logs below log traces
2025-08-27T22:43:41.416Z <#######> NSX 2362 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:54517), active=false) closed for the reason ERROR_WHILE_WRITING2025-08-27T22:43:41.416Z < NSX 2362 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Closing NettyConnection NettyConnection(NettyChannel(local=[#######>#######]:1235, remote=[#######]:54517), active=false)2025-08-27T22:43:41.417Z <#######>NSX 2362 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection closed received NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:54517), active=false)2025-08-27T22:43:41.417Z <#######> NSX 2362 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] tcp:CCP-[#######]: Unregistering accepted NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:54517), active=false) from its transport2025-08-27T22:43:41.417Z <#######> NSX 2362 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] Resolver asked to unregister NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:54517), active=false) that is not registered. Probably this connection never advertised a remote endpoint2025-08-27T22:43:41.417Z <#######> NSX 2362 - [nsx@6876 comp="nsx-manager" errorCode="MP101" level="ERROR" subcomp="ccp"] Closing connection NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:54517), active=false) because of unhandled exception io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.[Java stack trace omitted for brevity, as it's repetitive and lengthy]Caused by: java.security.cert.CertPathBuilderException: Unable to find certificate chain..
2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="INFO"] StreamSocket[64036] Open f:47 i:0 ? -> ssl://[#######]:1235] **on_connect 336151574-sslv3 alert certificate unknown** (SSL routines, ssl3_read_bytes)2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="7669825" level="WARNING"] StreamConnection[64036] Couldn't connect to 'ssl://[#######]:1235' **(error: 336151574-sslv3 alert certificate unknown** (SSL routines, ssl3_read_bytes))2025-08-27T22:43:42.094Z nsx-proxy[7669786] NSX #######- [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="7669825" level="WARNING"] RpcConnection[64036] Connecting to ssl://[#######]:1235 0] Couldn't connect to ssl://[#######]:1235 **(error: 336151574-sslv3 alert certificate unknown** (SSL routines, ssl3_read_bytes))2025-08-27T22:43:44.184Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2016" level="ERROR" errorCode="NET1111"] **Certificate validation failed: 18-self-signed certificate**.
2025-08-27T22:43:44.184Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="2016" level="ERROR" errorCode="NET4"] NetTransport[1] Accept on endpoint 'ssl://0.0.0.0:1234' failed with error 167772294-certificate verify failed (SSL routines) from remote endpoint 'ssl-tcp://[#######]:22075'2025-08-27T22:43:44.461Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] tcp:CCP-[#######]: Registering NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:41521), active=false) with its transport2025-08-27T22:43:44.494Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] **Client certificate not allow-listed: UID=[#######],CN=VMware-NSX-Host**,... for authType=RSA **failed: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain..**2025-08-27T22:43:44.494Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] NettyConnection(NettyChannel(local=[#######]:1235, remote=[#######]:41521), active=false) failed to complete SSL handshake2025-08-27T22:43:44.495Z < NSX #######>#######- [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="ccp"] error javax.net.ssl.SSLHandshakeException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
Environment
NSX
Cause
- The core problem is a mutual SSL/TLS certificate trust failure or Expired certificates.
- Neither the ESXi host nor the NSX Manager trusts the certificate presented by the other party during the Control Plane (CCP) communication handshake on port 1235.
- This means a secure communication channel (the CCP connection) cannot be established between the NSX Manager and the ESXi host.
Resolution
To resolve the issue, you need to re-establish the certificate trust between the NSX Manager and the ESXi Transport Node.
Procedure 1:
- Select the affected host and click “Host Disconnected” under NSX Configurations.
- A new window will open. Next to Configuration Complete, click “View Errors.”
- In the Select Errors to Resolve window, review the error and click “Resolve.”
- This process will re-sync the ESXi transport node certificate with the NSX Manager.
- Please note that the host will automatically enter NSX Maintenance Mode during this operation.
Procedure 2:
Push the ESXi transport node certificate to the NSX Manager node manually by following below instructions
- Obtain the NSX Manager thumbprint by logging into one of the MP nodes and execute the command below
NSX-MANAGER:> get certificate api thumbprint
- Once the thumbprint of the NSX manager is obtained, perform the host certificate push operation from the host using the command below
host-1# nsxcli -c push host-certificate <NSX Manager-IP/hostname> username admin thumbprint <cert-api-thumbprint-of-manager> password <NSX Manager password>
host-1# nsxcli -c sync-aph-certificates <NSX Manager-IP/hostname> username admin thumbprint <cert-api-thumbprint-of-manager> password <NSX Manager password>
Example:
[host-1]# nsxcli -c push host-certificate x.x.x.x username admin thumbprint <HEX> password ABCD!!
Host certificate was pushed to management plane successfully
Feedback
