Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

You see a critical alarm in the vSphere Client or vSphere Web Client for a Certificate expiry.
A CA Certificate that is in use in the environment is expiring or expired.
You have already renewed the certificates and have a new, valid CA Certificate in place. Remove expired old SSL certificate.
Attempts to remove the expired CA Certificate using the Web Client or other methods fail, and the Certificate is copied back to VMware Endpoint Certificate Store (VECS) after deletion.
Remove/delete trusted root certificate.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 6.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

Certificates are copied back to the VECS store because the CA Certificate which is expiring is published to the VMware Directory Service (VMDIR). When the Certificate is removed from VECS, VMDIR adds the Certificate back to VECS during a sync operation. This is done in order to ensure the integrity of the TRUSTED_ROOTS Certificate store, as deletion of an incorrect Certificate from this store could cause the environment to be irreparably damaged.

Resolution

To un-publish expired/expiring certificates from TRUSTED_ROOTS VECS Store:

List the certificates using vecs-cli.

On a vCenter Server Appliance, the path is here: /usr/lib/vmware-vmafd/bin
On a Windows vCenter Server, the path is here: C:\Program Files\VMware\vCenter Server\vmafdd

Run the following command:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli.exe entry list --store TRUSTED_ROOTS --text | more

Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
Find the Certificate you wish to remove and make a note of the Alias and the X509v3 Subject Key Identifier. In my example, it is as follows:
Alias : 2b724e6dd26e38b369a020f279f3bfc3369e2e7f
X509v3 Subject Key Identifier:
ED:CF:46:E5:CA:A6:8A:75:04:C0:D4:7B:2B:45:2C:08:53:10:F9:18

Note: There Could be several Certificates to remove. Any expired and not in use certificates should be removed to avoid certificate related alarms.
List the trusted certs published to the VMware Directory Service using the following command ([email protected] password required). This command is in the same location as vecs-cli:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert list

Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list

This will output a list of Certificates published to VMDIR. It will look similar to the following output:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli.exe trustedcert list
Enter password for [email protected]:
Number of certificates: 3
#1:
CN(id): EDCF46E5CAA68A7504C0D47B2B452C085310F918
Subject DN: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=psc1, OU=VMware
CRL present: yes
#2:
CN(id): 72B1C4C56A1A8A66B8C57182D551A29B78531ED0
Subject DN: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=psc2, OU=VMware
CRL present: yes
#3:
CN(id): 7AF0962806F5997107BF9A213E86DED4F853FF70
Subject DN: CN=CA, DC=vsphere, DC=local, C=US, ST=California, O=psc1, OU=VMware
CRL present: yes
Locate the Certificate's CN (thumbprint) which matches the Key Identifier from Step 2 above. In this example, the Certificate will be the first one in the list with the following CN:
EDCF46E5CAA68A7504C0D47B2B452C085310F918
Using the ID located in Step 4, run the following command, adjusting appropriately for your environment:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert get --id EDCF46E5CAA68A7504C0D47B2B452C085310F918 --login [email protected] --password <PASSWORD> --outcert C:\temp\oldcert.cer

Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert get --id EDCF46E5CAA68A7504C0D47B2B452C085310F918 --login [email protected] --password <PASSWORD> --outcert /tmp/oldcert.cer
Un-publish the CA Certificate from VMDIR by running the following command:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert unpublish --cert C:\temp\oldcert.cer

Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /tmp/oldcert.cer
Confirm that the Certificate was un-published by re-running the following command:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>dir-cli trustedcert list

Appliance:
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert list
Delete the Certificate from VECS utilizing the Alias located in Step 2 by running the following command:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli entry delete --store TRUSTED_ROOTS --alias 2b724e6dd26e38b369a020f279f3bfc3369e2e7f

Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 2b724e6dd26e38b369a020f279f3bfc3369e2e7f
Confirm that the Certificate was deleted by running the following command:
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli entry list --store TRUSTED_ROOTS --text | findstr Alias

Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias
Force a refresh of VECS by running the following command. This will ensure updates are pushed to the other PSCs in the environment if there is more than one.
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli force-refresh

Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
Confirm that the Certificate is no longer present. You can run the same command on all the PSCs to ensure the refresh done in Step 10 was successful.
Windows:
C:\Program Files\VMware\vCenter Server\vmafdd>vecs-cli entry list --store TRUSTED_ROOTS --text | findstr Alias

Appliance:
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep Alias
Restart all services on the PSCs and on the vCenter Servers and ensure that all services start and respond normally and that you can log in and manage the environment.

Additional Information

Note:

There might be certain situations were there are still older entries in TRUSTED_ROOTS that do not contain the certificate option "X509v3 Subject Key Identifier".
The best way to deal with this kind of scenario is to compare the CN(id) information of the other entries provided by dir-cli with the Subject Key Identifiers for those entries in TRUSTED_ROOTS which actually have them.
Once these have identified, you can compare the subject information for the remaining CN(id)s with the "Subject" option of the certificate in the VECS TRUSTED_ROOTS store to identify which one matches the certificate you want to remove from the store.

WARNING:

Proceed with EXTREME CAUTION. If the wrong Certificate is un-published and removed from VECS, this can damage the environment which can be irreparable.
Be absolutely certain that the Certificate you are removing is the correct Certificate to remove.
Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.

Mandatory precaution:

Ensure that all Platform Services Controllers in the federated environment are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems while powered off.
Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.