Verify and remove CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

Products

VMware vCenter Server

Issue/Introduction

Certificate Expiry and Removal Issues:

vCenter Server shows critical alarms in the vSphere Client indicating certificate expiry with messages stating:

'TRUSTED_ROOTS' expires on <date>

Certificate(s) in VECS TRUSTED_ROOTS store is about to expire
A CA certificate currently in use in the environment is expiring or has expired, and the same needs to be removed after installing a new certificate.
Attempts to remove expired CA certificates using the vSphere Client or other standard methods fail, with the certificate being automatically copied back to the VMware Endpoint Certificate Store (VECS) after deletion.

General Hygiene:

Need to remove an existing Trusted Root Certificate which is no longer needed on vCenter.

Common Impact: These certificate issues prevent proper certificate management, generate ongoing alarms or diagnostic warnings, and may interfere with maintenance operations or health assessments. While some scenarios may not immediately impact functionality, they indicate certificate store inconsistencies that should be resolved to maintain proper vCenter Server hygiene and avoid potential future complications during certificate operations.

Resolution

Verify the certificates that are expiring using the below command :

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep "Alias\|Not After\|Subject:\|Issuer:"

To un-publish expired/expiring certificates from TRUSTED_ROOTS VECS Store:

List the certificates using vecs-cli.

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less

Find the certificate that needs to be removed and make a note of the Alias.

Example:
```
Alias : ####################################2e7f
```
Note: There could be several certificates to remove. Any expired and not in use certificates should be removed to avoid certificate related alarms.

Using the Alias ID located in Step 2, run the following command to save the certificate to /root/ folder, adjusting appropriately for the environment:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store TRUSTED_ROOTS --alias ####################################2e7f --output /root/<aliasID>.cer

Unpublish the expired/expiring CA certificate from VMDIR, it will prompt for SSO Administrator credentials.
```
/usr/lib/vmware-vmafd/bin/dir-cli trustedcert unpublish --cert /root/<aliasID>.cer
```
Delete the certificate from VECS utilizing the Alias located in Step 2:
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store TRUSTED_ROOTS --alias ####################################2e7f
```
Notes:
If the alias has special characters, enclose the entire alias in single quotes when using the vecs-cli command for eg. --alias 'https://[IP]:9997/vasa'
If the command is failing with error "Operation failed with error ERROR_OBJECT_NOT_FOUND", ignore the error and proceed further. This error will be logged if the certificate is already removed from the store as part of Step 4.
Perform force refresh of VECS to sync certificate from VMDIR.
```
/usr/lib/vmware-vmafd/bin/vecs-cli force-refresh
```
Confirm that the certificate is no longer present.

Note: Output of this command should not be listing the Alias ID that was removed in above steps.
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep <aliasID>
```
Restart all services on the vCenter Servers, ensuring that all services start and respond normally, and that login and management of the environment are functioning properly.

service-control --stop --all
service-control --start --all

Additional Information

WARNING:

Be absolutely certain that the certificate that is being removed is the correct certificate to remove. Un-publishing wrong certificate can impact the environment.
Verify if the root certificate that needs to be removed is not used by the vCenter. To verify follow below steps:
- Make a note of the alias of the certificate that needs to be removed captured in step 2 of the resolution section
- Capture the "Subject Key Identifier" information for the target root certificate from below command
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
```
- The "Subject Key Identifier" will be marked as "Authority Key Identifier" in the Machine_SSL certificate details. It can be fetched and verified by using below command. If you find it, which means that the vCenter is currently using the root certificate
```
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text
```
Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.

Mandatory precaution:

Ensure that all vCenter Servers the federated environment (ELM) are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the vCenter Servrs when the snapshot operation is complete.
Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.