Preparing ESXi host with NSX-T fails with an error "Waiting for connection to Managers"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You're running NSX-T Data Center 3.x or later
• Preparing ESXi host as NSX Transport Node fails and UI shows following error "Waiting for connection to Managers".
• Attempting to resolve the installation from NSX UI it further fails with error "Failed to install software on host. Time out waiting for host to join NSX Manager".
• Host will have all the NSX VIBs but still the preparation fails.
• You may observe certificate related errors to Heartbeat related logs in /var/run/log/nsx-syslog.log:

Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="8140587" level="WARNING"] StreamConnection[5 Connecting to ssl://<nsx-mngr-ip>:1235 sid:5] Couldn't connect to 'ssl://<nsx-mngr-ip>:1235' (error: 336151574-sslv3 alert certificate unknown)
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="8140587" level="WARNING"] StreamConnection[5 Error to ssl://<nsx-mngr-ip>:1235 sid:-1] Error 336151574-sslv3 alert certificate unknown
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="8140587" level="WARNING"] RpcConnection[5 Connecting to ssl://<nsx-mngr-ip>:1235 0] Couldn't connect to ssl://<nsx-mngr-ip>:1235 (error: 336151574-sslv3 alert certificate unknown)
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="8140587" level="WARNING"] RpcTransport[0] Unable to connect to ssl://<nsx-mngr-ip>:1235: 336151574-sslv3 alert certificate unknown

• You may observe logs indicating that NSX manager is waiting for heartbeat status under /var/log/proton/nsxapi.log:

INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.start
...
INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.start
...
INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.validating_dependencies
...
INFO ActivityWorkerPool-1-9 Esx60SfdmManager 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Waiting for heartbeat status for host <uuid>, number of tries 48
...

• ​​​​​​You may observe below entry in the NSX Manager log /var/log/vmware/appl-proxy-rpc.log.

MDC-NSXTMGR03 NSX 96476 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="96480" level="ERROR" errorCode="NET1111"] Certificate validation failed: 9-certificate is not yet valid#012Certificate:#012 Data:#012
...
Not Before: Aug 16 17:07:41 2023 GMT#012 Not After : Nov 16 17:07:41 2025 GMT#012 

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center
VMware NSX-T Data Center 4.x

Cause

Time skew between ESXi host & NSX-T Manager caused by time sync issue results in certificate validation failure which leads to communication failure between Management Plane and Host.
As per the log entry above
Not Before: Aug 16 14:06:41 2023 GMT#012 Not After : Sep 14 14:06:41 2025 GMT#012
We see that the certificate was not valid yet at the time of the host preparation, i.e, the host is being prepared before the certificate became valid.

Resolution

Please make sure that NTP is configured properly on both NSX-T Managers and ESXi hosts and ensure the time is in sync between them.

Additional Information

Impact/Risks:
Inability to prepare ESXi host as NSX-T Transport Node.