ESXi host is disconnected on startup

Products

• VMware NSX

Issue/Introduction

• TN Certificate was replaced with following API.
  POST api/v1/trust-management/certificates/action/replace-host-certificate/<TN_UUID>


• The API request had only 2 '\n'.
  Exapmle request.json:
  {
    "display_name": "cert_sample",
    "pem_encoded": "-----BEGIN CERTIFICATE-----\n<cert>\n-----END CERTIFICATE-----",
    "private_key": "-----BEGIN PRIVATE KEY-----\n<private_key>\n-----END PRIVATE KEY-----"
}
  
  In this example above, '\n' appears only twice.
  
  
• After the certificate was replaced by POST api/v1/trust-management/certificates/action/replace-host-certificate/<TN_UUID>, counting line by wc command on ESXi host shows the certificate "host-cert.pem" has only 2 or less.
  wc -l /etc/vmware/nsx/host-cert.pem
2 /etc/vmware/nsx/host-cert.pem
  
  
• nsx-proxy service is started. 
  Following operations also start nsx-proxy.
  
  • Restart nsx-proxy with /etc/init.d/nsx-proxy restart
  • Reboot ESXi host
  • Upgrade ESXi host
    
    
• nsx-syslog shows nsx-proxy fails to connect to NSX Manager on ESXi /var/log/nsx-syslog.log
  <TIMESTAMP> Wa(180) nsx-proxy[<PID>]: NSX <PID> - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="<TID>" level="WARNING"] RpcConnection[522067 Connecting to ssl://<NSX_MANAGER>:1235 0] Couldn't connect to ssl://<NSX_MANAGER>:1235 (error: 336151574-sslv3 alert certificate unknown)
  
  
• syslog shows nsx-proxy generates a new certificate and deletes the existing certificate at /etc/vmware/nsx/host-cert.pem. syslog is found at ESXi /var/log/syslog.
  <TIMESTAMP> In(<INT>) NSX[<PID>]: Generating the certificate
<TIMESTAMP> In(<INT>) NSX[<PID>]: Generating host certificate with TN uuid = .
<TIMESTAMP> In(<INT>) NSX[<PID>]: Deleting the /etc/vmware/nsx/host-privkey.pem and /etc/vmware/nsx/host-cert.pem.
  
  
• The certificate is located at /etc/vmware/nsx/host-cert.pem. Verify "Not Before" is when nsx-proxy replaces the certificate.
  openssl x509 -in /etc/vmware/nsx/host-cert.pem -text -noout
          Validity
              Not Before: <TIMESTAMP>

Environment

NSX 4.x

Cause

nsx-proxy validates the host certificate if it has more than 2 lines with wc command.
If there are only 2 lines or less, nsx-proxy generates a new certificate and replaces the original certificate.

Because the newly generated certificate is not trusted by NSX Manager, the connection from ESXi host is rejected by NSX Manager.

Resolution

To restore from disconnected state, push host certificate from ESXi host to NSX Manager.

1. Identify NSX Manager thumbprint by logging into the NSX Manager as admin: get certificate api thumbprint
2. To push the new cert-key pair to the Manager, from the root user on the Host, run:
   nsxcli -c "push host-certificate <Manager hostname-or-IP> username admin thumbprint <thumbprint from step 1>"