NSX Vibs fails on Transport Node with error: Could not find a trusted signer: self-signed certificate

Products

• VMware NSX

Issue/Introduction

• In the NSX Manager, you see the following error in /var/log/proton/nsxapi.log:
  <timestamp> <NSX_Name> NSX 2876 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] checkServerTrusted: ############,CN=#######,OU=##,O=##,L=##,C=## for authType=ECDHE_RSA failed: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.
  
  
• In the ESXi host, you may see the following error in /var/log/esxupdate.log:
  <timestamp> Db(15) esxupdate[ID] Verifying VIB VMware_bootbank_nsx-python-protobuf_2.6.1-19195979 signature #1
<timestamp> Er(11) esxupdate[ID] Failed to verify VIB signature #2: ('VMware_bootbank_nsx-python-protobuf_2.6.1-19195979', 'Could not find a trusted signer: self-signed certificate')

Cause

During the NSX vib installation, the ESXI host validates its signature to make sure the vib is from a trusted source. It does this process by validating the trusted signature against its trusted chain. The failure of this issue can be due to several reasons:

1. Expired vCenter Certificates - VMware Certificate Authority (VMCA) in the vCenter acts as a central Certificate Authority (CA) or at least a trust anchor for its managed ESXi hosts. ESXi hosts establish a secure, authenticated connection with vCenter using certificates. 
   
   
2. Time synchronization issues - If the ESXi host's time is significantly out of sync with vCenter or the certificate's validity period, it can cause the certificate to be perceived as "not yet valid" or "expired," even if it isn't, leading to trust validation failures.
   
   
3. Corrupted ESXi Trust Store - If the ESXi host has a custom certificate, it is possible can have a malformed trusted store /etc/vmware/ssl/castore.pem
   
   
4. Expired Transport Node Certificate - Each ESXi host has a Transport Node certificate located in /etc/vmware/nsx/host-cert.pem.

Resolution

1. Check if the Machine_SSL or STS certificates in the vCenter are expired or if there is a trusted mismatch. Resolve certificate issues with the vCenter Server and then retry the vibs installation. Refer to - vCert - Expired Certificate Replacement Script
   
   
2. Make sure that NSX Managers, vCenter, and ESXi hosts have time synchronization correctly. For better time synchronization, please use the same NTP server on all components.
   
   
3. The following command will list the trusted store certificates in the ESXi host:
   openssl x509 -in /etc/vmware/nsx/host-cert.pem -text -noout
   In case of a malformed certificate, please open a case with Broadcom for further assistance
   
   
4. In case of Transport Node Expired Certificate, please refer to Alarm For Transport Node Certificate Has Expired.