Preparing ESXi host with NSX fails with an error "Waiting for connection to Managers"

Products

• VMware NSX

Issue/Introduction

• You're running NSX-T Data Center 3.x or later
• Preparing ESXi host as NSX Transport Node fails and UI shows following error "Waiting for connection to Managers".
• Attempting to resolve the installation from NSX UI and it fails with error "Failed to install software on host. Time out waiting for host to join NSX Manager".
• Host will have all the NSX VIBs but still the preparation fails.
• You may observe certificate related error logs in /var/run/log/nsx-syslog.log:

Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="8140587" level="WARNING"] StreamConnection[5 Connecting to ssl://<nsx-mngr-ip>:1235 sid:5] Couldn't connect to 'ssl://<nsx-mngr-ip>:1235' (error: 336151574-sslv3 alert certificate unknown)
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="8140587" level="WARNING"] StreamConnection[5 Error to ssl://<nsx-mngr-ip>:1235 sid:-1] Error 336151574-sslv3 alert certificate unknown
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="8140587" level="WARNING"] RpcConnection[5 Connecting to ssl://<nsx-mngr-ip>:1235 0] Couldn't connect to ssl://<nsx-mngr-ip>:1235 (error: 336151574-sslv3 alert certificate unknown)
Wa(180) nsx-proxy[8140557]: NSX 8140557 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="8140587" level="WARNING"] RpcTransport[0] Unable to connect to ssl://<nsx-mngr-ip>:1235: 336151574-sslv3 alert certificate unknown

• You may observe logs indicating that NSX manager is waiting for heartbeat status under /var/log/proton/nsxapi.log:
  

  INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.start
  ...
  INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.start
  ...
  INFO ActivityWorkerPool-1-4 HostTNDeploymentProgressServiceImpl 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] FN Operation: INSTALL. Host: <uuid> current progress percentage: 0, stateDescription: deployment.progress.fn.validating_dependencies
  ...
  INFO ActivityWorkerPool-1-9 Esx60SfdmManager 4878 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Waiting for heartbeat status for host <uuid>, number of tries 48
  ...

• ​​​​​​You may observe below entry in the NSX Manager log /var/log/vmware/appl-proxy-rpc.log.
  

  MDC-NSXTMGR03 NSX 96476 - [nsx@6876 comp="nsx-manager" subcomp="appl-proxy" s2comp="nsx-net" tid="96480" level="ERROR" errorCode="NET1111"] Certificate validation failed: 9-certificate is not yet valid#012Certificate:#012 Data:#012
  ...
  Not Before: Aug 16 17:07:41 2023 GMT#012 Not After : Nov 16 17:07:41 2025 GMT#012 

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4x

Cause

A Time skew between ESXi host & NSX-T Manager caused by time sync issue results in certificate validation failure which leads to communication failure between Management Plane and Host.
As per the log entry below

Not Before: Aug 16 14:06:41 2023 GMT#012 Not After : Sep 14 14:06:41 2025 GMT#012

We see that the certificate was not valid yet at the time of the host preparation, i.e., the host is being prepared before the certificate became valid.

Resolution

This is a known issue impacting VMware NSX.

 

 

Workaround:

• Check that NTP is configured properly on both NSX-T Managers and the ESXi hosts
• Ensure the time is in sync between both.
  
  

For more information regarding NTP configuration, please review Configuring NTP on Appliances and Transport Nodes

Additional Information

Impact/Risks:
Inability to prepare ESXi host as NSX-T Transport Node.