ESXi host root and vcf-svc accounts disconnected for hosts managed by workload domain vCenter and cannot be remediated

Products

• VMware SDDC Manager

Issue/Introduction

A password remediation attempt for either the root or vcf-svc account on the affect ESXi host results in an error -

Message: Validation of password of ESXi host: host_fqdn has failed

Remediation Message: Check if the password used to connect to ESXi host is valid

Cause: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

 

The password is known to be good, having been retrieved from the lookup_passwords on the SDDC Manager and used to successfully SSH to the affected ESXi host.

 

The VMCA certificate mode is expected to be vmca, and has been confirmed to be vmca per the below -

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-security-8-0/securing-esxi-hosts/certificate-management-for-esxi-hosts/change-the-certificate-mode.html 

 

An error similar to the below can be seen in the SDDC Manager log /var/log/vmware/vcf/operationsmanager/operationsmanager.log - 

YYY-MM-DDThh:mm:ss.sss+0000 DEBUG [vcf_om,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.v.s.t.DynamicTrustManager,om-exec-16] Error checking certificate chain [email protected], CN=host_fqdn, OU=VMware Engineering, O=VMware, L=Palo Alto, ST=California, C=US for validity.
java.security.cert.CertificateException: No issuer certificate for certificate in certification path found.

 

There has been recent certificate renewal activity on management appliances (e.g. SDDC Manager, management vCenter, workload domain vCenter) in the environment that may have introduced the issue.

Environment

VCF 8.x

Cause

The workload domain vCenters current VMCA root certificate does not exist in the SDDC Managers trusted certificates store and/or cacerts store.

This can be verified as follows on the SDDC Manager -

pass=$(cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key)

keytool -list -v -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass $pass | less

The current VMCA root certificate will not appear in the output.

 

 

 

Resolution

Import the vCenters current VMCA root certificate into the SDDC Manager trusted certificates store and cacerts store by leveraging the article below -

https://knowledge.broadcom.com/external/article/316007/how-to-import-the-vcenter-root-certifica.html