Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

Products

VMware vCenter Server

Issue/Introduction

When you encounter certificate-related issues in your vCenter Server environment, you may experience one or more of the following scenarios:

Certificate Expiry and Removal Issues:

You receive critical alarms in the vSphere Client indicating certificate expiry with messages stating:

'TRUSTED_ROOTS' expires on <date>

A CA certificate currently in use in your environment is expiring or has expired, and you need to remove the old certificate after installing a new certificate.
Your attempts to remove expired CA certificates using the vSphere Client or other standard methods fail, with the certificate being automatically copied back to the VMware Endpoint Certificate Store (VECS) after deletion

General hygiene:

That you want to remove/delete an existing trusted root certificate which is no longer needed

Common Impact: These certificate issues prevent proper certificate management, generate ongoing alarms or diagnostic warnings, and may interfere with maintenance operations or health assessments. While some scenarios may not immediately impact functionality, they indicate certificate store inconsistencies that should be resolved to maintain proper vCenter Server hygiene and avoid potential future complications during certificate operations.

Environment

VMware vCenter Server 6.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Resolution

To un-publish expired/expiring certificates from TRUSTED_ROOTS VECS Store:

List the certificates using vecs-cli.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

Find the certificate that needs to be removed and make a note of the Alias.

Example:
```
Alias : 2b724e6dd2####################c3369e2e7f
```
Note: There could be several certificates to remove. Any expired and not in use certificates should be removed to avoid certificate related alarms.
Using the Alias ID located in Step 2, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment:
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias 2b724e6dd2########c3369e2e7f --output /root/<aliasID>.cer
```
Unpublish the expired/expiring CA certificate from VMDIR, it will prompt for SSO Administrator credentials.
```
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer
```
Delete the certificate from VECS utilizing the Alias located in Step 2:
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias 2b724e6dd2####################c3369e2e7f
```
Notes:
If the alias has special characters, enclose the entire alias in single quotes when using the vecs-cli command for eg. --alias 'https://[IP]:9997/vasa'
If the command is failing with error "Operation failed with error ERROR_OBJECT_NOT_FOUND", ignore the error and proceed further. This error will be logged if the certificate is already removed from the store as part of Step 4.
Perform force refresh of VECS to sync certificate from VMDIR.
```
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```
Confirm that the certificate is no longer present.

Note: Output of this command should not be listing the Alias ID that was removed in above steps.
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep <aliasID>
```
Restart all services on the vCenter Servers, ensuring that all services start and respond normally, and that login and management of the environment are functioning properly.

service-control --stop --all
service-control --start --all

Additional Information

WARNING:

Proceed with EXTREME CAUTION. If the wrong Certificate is un-published and removed from VECS, this can damage the environment which can be irreparable.
Be absolutely certain that the certificate that is being removed is the correct certificate to remove.
Verify if the root certificate that you are planning to remove is not used by the vCenter. To verify follow below steps
- Make a note of the alias of the certificate that needs to be removed captured in step 2 of the resolution section
- capture the "Subject Key Identifier" information for the target root certificate from below command
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
```
- The "Subject Key Identifier" will be marked as "Authority Key Identifier" in the Machine_SSL certificate details. It can be fetched and verified by using below command. If you find it, which means that the vCenter is currently using the root certificate
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text
```
Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.

Mandatory precaution:

Ensure that all vCenter Servers the federated environment (ELM) are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the vCenter Servrs when the snapshot operation is complete.
Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.