Error: certificate_unknown(46) - ESXi host root and vcf-svc accounts disconnected in VCF workload domain

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• Attempting a password remediation or rotation for the root or vcf-svc account on an ESXi host via SDDC Manager fails.
• The task fails with one of the following validation and remediation messages:
  
  Message: Validation of password of ESXi host: <HOST_FQDN> has failed
Remediation Message: Check if the password used to connect to ESXi host is valid
Cause: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

Cause: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
  
  
• Despite the error message suggesting an invalid password, the password is confirmed to be correct. Successfully retrieving the password using the lookup_passwords utility on the SDDC Manager allows for a successful direct SSH connection to the affected ESXi host.
• Reviewing the /var/log/vmware/vcf/domainmanager/domainmanager.log or operationsmanager.logreveals one of the following underlying SSL/TLS certificate exceptions:
  
  YYY-MM-DDThh:mm:ss.sss+0000 DEBUG [vcf_om,###############################,####] [c.v.v.s.t.DynamicTrustManager,om-exec-16] Error checking certificate chain EMAILADDRESS=[e-mail], CN=<HOST_FQDN>, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US for validity.
java.security.cert.CertificateException: No issuer certificate for certificate in certification path found.

Cause: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
  
  
• The ESXi host certificate mode is verified to be set to vmca, and has been confirmed to be vmca per the below article-
  Change the ESXi Certificate Mode

• There has been recent certificate renewal activity on management appliances (e.g. SDDC Manager, management vCenter, workload domain vCenter) in the environment that may have introduced the issue.

Environment

• VCF 5.x
• VCF 9.x

Cause

The workload domain vCenters current VMCA root certificate does not exist in the SDDC Managers trusted certificates store and/or cacerts store.

This can be verified as follows on the SDDC Manager -

1. Log in to the SDDC Manager as root.
2. Retrieve the password: pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)
3. List the keystore content: keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass | less

The current VMCA root certificate will not appear in the output.

Resolution

To resolve this issue and re-establish trust, you must import the vCenter Server's active VMCA root certificate into both the SDDC Manager's trusted certificates store and the Java cacerts keystore.

For step-by-step instructions on how to perform this import, refer to the following article:

Import the vCenter root certificate into the SDDC manager TrustStore