certificate-manager failed at 0% because there are too many entries in TRUSTED_ROOT_CRLS

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

The certificate renewal process initiated by the user using the certificate-manager command failed at 0% completion.
In /var/log/vmware/vmcad/certificate-manager.log, there are error messages about deleting entry from certificate store:

Error in deleting entry __MACHINE_CERT from VECS store MACHINE_SSL_CERT.

Environment

vCenter 7.x
vCenter 8.x

Cause

Upon reviewing the certificate stores, a significant number of entries were identified in the TRUSTED_ROOT_CRLS store:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS | grep Number
​​​​​Number of entries in store :    3165

 

Resolution

Note: Before proceeding with the steps below, create a snapshot of the vCenter Server.

Standalone vCenter Server: Take a non-memory snapshot. 

Linked Mode vCenter Servers: 
1. Power off all vCenter VMs in Linked Mode.
2. Take snapshots of each VM.
3. Power on the vCenter VMs. 

To remove the extra entries in TRUSTED_ROOT_CRLS  store:

1. Copy the below script and save: vi clean_trusted_root_crl_store.sh

#!/bin/bash
cd /etc/ssl/certs
mkdir /tmp/pems
mkdir /tmp/OLD-CRLS-CAs
mv *.pem /tmp/pems && mv *.* /tmp/OLD-CRLS-CAs
h=$(/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOT_CRLS --text | grep Alias | cut -d : -f 2)
for hh in "echo "${h[@]}"";do echo "Y" | /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOT_CRLS --alias $hh;done
mv /tmp/pems/* .
for l in `ls *.pem`;do ln -s $l ${l/pem/0};done
service-control --stop vmafdd && service-control --start vmafdd

2. Make the script executable: chmod +x clean_trusted_root_crl_store.sh
3. Run the script: ./clean_trusted_root_crl_store.sh