"A general system error occurred: Unable to push CA certificates and CRLs to host" when attempting to add an ESXi host to vCenter or when attempting to renew the certificate for the host

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

Attempting to add or reconnect an ESXi host to vCenter fails with the following error:

"A general system error occurred: Unable to push CA certificates and CRLs to host"

The same error may be observed when attempting to renew the certificate of an ESXi host from vCenter using the vSphere Web Client or the host client. In the /var/run/log/hostd.log file on the ESXi host, the following entries may be observed:
- [YYYY-MM-DDTHH:MM:SS] error hostd[2099958] [Originator@6876 sub=Vimsvc.CertMgr opID=xxxxxxx-86e0-4120-a9c5-xxxxxx-122905-auto-122907-h5c:70009449-db-6fa0 user=vpxuser:VSPHERE.LOCAL\administrador] Certificate is not a valid CA certificate:
  -----BEGIN CERTIFICATE-----
  <certificate string>
  --> -----END CERTIFICATE-----
  
  or
- Task Created : haTask--vim.host.CertificateManager.replaceCACertificatesAndCRLs-xxxxx
  [YYYY-MM-DDTHH:MM:SS] info hostd[3597154] [Originator@6876 sub=Vimsvc.CertMgr opID=m5f9ixec-#######-auto-#####-h5:########-fc-#### user=vpxuser:EXAMPLE_DOMAIN\EXAMPLE_USER] Discarding non-CA certificate:
  -----BEGIN CERTIFICATE-----
  --> RAW CERT STRING
  -----END CERTIFICATE-----

Environment

VMware vSphere ESXi 6.7

VMware vSphere ESXi 7.x

Cause

This issue occurs due to self-signed or Non-CA certificates in the TRUSTED_ROOTS store on the vCenter Server getting pushed to the ESXi host when adding/reconnecting the host to vCenter or when renewing the Certificate on the host.

Resolution

Note: This issue may occur due to time sync issues, if the time on ESXi host is out of sync with the time on the vCenter Server. Please verify that the time on the ESXi host and vCenter are in sync before proceeding with the steps below.

To resolve the issue, perform one of the options below:

Option 1:
Remove the Non-CA or Self-Signed certificates from the TRUSTED_ROOTS store of vCenter Server by following the steps below.

Caution: In an environment with a VASA provider (eg - VVOL) configuration, it has been observed that the VASA provider self-signed certificates are getting added to the TRUSTED_ROOTS store and removing these certificates from TRUSTED_ROOTS will result in VASA provider becoming Offline. If there is such configuration:

Proceed with Option 2 (this option is only applicable if the ESXi host is on version 6.7 Update 3 or later)
Contact the VASA provider vendor to configure the signed certificate before proceeding to Option 1

Execute the command below to list the 'Key Usage' and 'Alias' field of each certificate in the TRUSTED_ROOTS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Key Usage' -A 1 | grep -v "Entry type"

Identify the Certificate Aliases that does not have the 'Certificate Signing' keyword under the Key Usage parameter, these are Non-CA certificates

In the example below, the Alias 5de##########ddad3d4c38447210f########## does not have a Certificate Signing which means that it is not a CA Certificate:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Key Usage' -A 1 | grep -v "Entry type"

Alias : e74##########c2c6a44312962be############
--
X509v3 Key Usage: critical
Certificate Sign, CRL Sign

Alias : 5de##########ddad3d4c38447210f##########
--
X509v3 Key Usage:
Digital Signature, Key Encipherment, Key Agreement

Take a backup of the non-CA certificates as identified from the steps above by executing below command. This step and the following steps needs to be performed for each certificate identified from the steps above.

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias <alias name> --output /var/tmp/<aliasname.crt>

Example:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 5de##########ddad3d4c38447210f########## --output /var/tmp/55de##########ddad3d4c38447210f##########.crt

Un-publish the certificate from the VMware Directory Service by executing the command below, it will ask for the SSO Administrator password.

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert <certificate full path> --login administrator

Note: If it is not published, the following error will be observed "dir-cli failed. Error 1168: Operation failed with error ERROR_NOT_FOUND (1168)".

- Please ignore this message and proceed with the next step to remove the certificate from the store.

Example:

/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /var/tmp/5de##########ddad3d4c38447210f##########.crt --login administrator

dir-cli unpublish command should automatically delete the certificate from the TRUSTED_ROOTS store if it is published. If not, execute the command below to remove the certificate from the VECS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias <alias name> -y

Example:

/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 5de##########ddad3d4c38447210f########## -y

Reattempt to add the ESXi host to vCenter Server or reattempt to renew the certificate

Option 2:
Modify the ESXi advanced option "Config.HostAgent.ssl.keyStore.allowSelfSigned" introduced in ESXi 6.7 Update 3 to ignore the Self Signed Certificates.

Connect to the ESXi using the Host Client
Select Manage Tab
Select Advanced Settings
Locate the option "Config.HostAgent.ssl.keyStore.allowSelfSigned"
Change the value from false to true
Reboot the ESXi host.
Retry adding the ESXi host to vCenter Server or reattempt to renew the certificate.

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Please note that advanced setting "Config.HostAgent.ssl.keyStore.allowSelfSigned" is only available in ESXI 6.7 U3 and later, therefore it is a requirement to be at this release.

Impact/Risks:

This change allows for self-signed certificate to be added to the ESXi.