A general system error occurred: Unable to push CA certificates and CRLs to host, Adding or Reconnecting 6.7 ESXi host to vCenter Server fails
search cancel

A general system error occurred: Unable to push CA certificates and CRLs to host, Adding or Reconnecting 6.7 ESXi host to vCenter Server fails

book

Article ID: 312677

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Adding or Reconnecting ESXi 6.7 host fails with below error message
"A general system error occurred: Unable to push CA certificates and CRLs to host"
  • Same error is observed during Renewing the certificate of ESXi host from vCenter Server using WebClient or UI Client
  • In /var/run/log/hostd.log file on the ESXi host, you see entries similar to :
2019-09-12T11:39:17.594Z error hostd[2099958] [Originator@6876 sub=Vimsvc.CertMgr opID=22aa0399-86e0-4120-a9c5-43173a148302-122905-auto-122907-h5
c:70009449-db-6fa0 user=vpxuser:VSPHERE.LOCAL\administrador] Certificate is not a valid CA certificate: -----BEGIN CERTIFICATE-----
<certificate string>
--> -----END CERTIFICATE-----


Environment

VMware vSphere ESXi 6.7

Cause


This issue is due to self-signed or Non-CA certificates in TRUSTED_ROOTS store on the vCenter Server getting pushed to ESXi host while adding/reconnecting or while renewing the Certificate on the host.

Resolution


Note: This issue can also happen due to time sync issues, if the ESXi host time is incorrect when compared to vCenter Server. Please verify the time on ESXi and vCenter Server before proceeding with below steps.

To resolve the issue perform one of below Options :

Option 1:
Remove Non-CA or Self-Signed certificates from TRUSTED_ROOTS store of vCenter Server by following below steps.
 
Caution: In environment with VASA provider (eg - VVOL) configuration, it is observed that VASA provider self-signed certificates are getting added to TRUSTED_ROOTS store and removing these certificates from TRUSTED_ROOTS will result in VASA provider becoming Offline. If you have such configuration :
  • Proceed with Option 2 if ESXi Host build is 6.7 Update 3 or later 
  • Contact VASA provider vendor to configure signed certificate before proceeding to Option 1
 
  • Execute below command to list Key Usage and Alias field of each certificate in TRUSTED_ROOTS store
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Key Usage' -A 1 | grep -v "Entry type"
  • Identify the Certificate Aliases which does not have 'Certificate Signing' keyword under Key Usage parameter, these are Non-CA certificates
In below example, Alias 5deae7b5e95deddad3d4c38447210ffe23119d6d does not have Certificate Signing which means it is not a CA Certificate 

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Key Usage' -A 1 | grep -v "Entry type"
Alias : e74a421e72fa9c2c6a44312962be982ba1c251bf
--
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign

Alias : 5deae7b5e95deddad3d4c38447210ffe23119d6d
--
            X509v3 Key Usage:
                Digital Signature, Key Encipherment, Key Agreement
  • Take backup of Non-CA certificates identified from above steps by executing below command. This step and following steps needs to be performed for each certificate identified from above step.
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias name> --output /var/tmp/<aliasname.crt>
 
Example:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 5deae7b5e95deddad3d4c38447210ffe23119d6d --output /var/tmp/5deae7b5e95deddad3d4c38447210ffe23119d6d.crt
  • Un-publish the certificate from VMware Directory Service by executing below command, it will ask for SSO Administrator password
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <certificate full path> --login administrator

Note: If it is not published, you will observe error "dir-cli failed. Error 1168: Operation failed with error ERROR_NOT_FOUND (1168)". Please ignore this message and proceed with next step to remove the certificate from the store.

Example:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /var/tmp/5deae7b5e95deddad3d4c38447210ffe23119d6d.crt --login administrator
  • dir-cli unpublish command should automatically delete the certificate from TRUSTED_ROOTS store if it is published. If not, execute below command to remove the certificate from VECS store
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <alias name> -y

Example:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 5deae7b5e95deddad3d4c38447210ffe23119d6d -y
  • Retry adding the ESXi host to vCenter Server or certificate renew operation

Option 2:
Modify the advanced configuration "Config.HostAgent.ssl.keyStore.allowSelfSigned" introduced in ESXi 6.7 Update 3 to ignore the Self Signed Certificates. 
  • Connect to the ESXi using Host Client
  • Select Manage Tab
  • Select Advanced Settings 
  • Locate the option "Config.HostAgent.ssl.keyStore.allowSelfSigned"
  • Edit the value from false to true 
  • Reboot the ESXi host.
  • Retry adding the ESXi host to vCenter Server or certificate renew operation


Additional Information


VMware Skyline Health Diagnostics for vSphere - FAQ
Please note that advanced setting "Config.HostAgent.ssl.keyStore.allowSelfSigned" is only available in ESXI 6.7 U3 and above, therefore it is a requirement to be at this release.

Impact/Risks:

This change allows for self-signed certificate to added to the ESXi

Powered by Wolken Software