Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS) in vCSA using script

Products

• VMware vCenter Server

Issue/Introduction

Remove certificates from the TRUSTED_ROOTS store in a streamlined way

Symptoms:

• You see a critical alarm in the vSphere Client or vSphere Web Client for a Certificate expiry.
• A CA Certificate that is in use in the environment is expiring or expired.
• You have already renewed the certificates and have a new, valid CA Certificate in place. Remove expired old SSL certificate.
• Attempts to remove the expired CA Certificate using the Web Client or other methods fail, and the Certificate is copied back to VMware Endpoint Certificate Store (VECS) after deletion.
• Remove/delete trusted root certificate.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

• Certificates are copied back to the VECS store because the CA Certificate which is expiring is published to the VMware Directory Service (VMDIR).
• When the Certificate is removed from VECS, VMDIR adds the Certificate back to VECS during a sync operation.
• This is done in order to ensure the integrity of the TRUSTED_ROOTS Certificate store, as deletion of an incorrect Certificate from this store could cause the environment to be irreparably damaged.

Resolution

Note: Make sure you take snapshot of the VCSA VM before making any changes. 

 

To un-publish expired/expiring/unwanted certificates from TRUSTED_ROOTS VECS Store:

1. Download the attached removeroot.sh script attached to this article
2. Upload the attached script to the VCSA with embedded PSC in the /tmp folder, or copy its contents to a text file on the appliance using vi

Note: You may use WinSCP to upload the script to VCSA. For additional information, see Error when uploading files to vCenter Server Appliance using WinSCP (2107727) .

If you get an error for connecting to the VCSA via WinSCP run the following command:

• • chsh -s /bin/bash root as per above link.

3. cd to /tmp folder
4. Run chmod +x removeroot.sh to make the file executable. 
5. Run ./removeroot.sh

6. Enter the number of the certificate you want to remove from the list above, e.g. 2, then enter the administrator password for your SSO domain

7. Restart all services on the PSCs and on the vCenter Servers and ensure that all services start and respond normally and that you can log in and manage the environment.

# service-control --stop --all
# service-control --start --all

        KB reference: Stopping, Starting or Restarting VMware vCenter Server Appliance services

Additional Information

An alternate scripted method (vCert) of removing expired or unused certificates can be found here:

https://knowledge.broadcom.com/external/article/385107/vcert-scripted-vcenter-expired-certific.html

     - For removal from vmdir use the menu navigation "3. Manage Certificates" --->  "3.  CA Certificates in VMware Directory"

     - For removal from VECS use the menu navigation  "3. Manage Certificates" --->  "4. CA Certificates in VECs Directory"

Manual method to remove expired or unused certificates from trusted root certificates:

• Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS) (2146011)
  • If SubjectKey is blank the scripted method will fail and the manual method must be used. See Error: "dir-cli failed. Error 87: Operation failed with error ERROR_INVALID_PARAMETER (87)" and "Error 1168: Operation failed with error ERROR_NOT_FOUND (1168)" using removeroot.sh for more details.

• Refer to KB CertificateStatusAlarm - There are certificates that expired or about to expire / Certificate Status Change Alarm Triggered on VMware vCenter Server for more information on removing expired certificates from other certificate stores.

• VMware Skyline Health Diagnostics for vSphere - FAQ

Impact/Risks:

WARNING:

• Proceed with EXTREME CAUTION. If the wrong Certificate is un-published and removed from VECS, this can damage the environment which can be irreparable.
• Be absolutely certain that the Certificate you are removing is the correct Certificate to remove.
• Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.

• Verify if the root certificate that needs to be removed is not used by the vCenter. To verify follow below steps:
  • Make a note of the alias of the certificate that needs to be removed captured in step 2 of the resolution section
  • Capture the "Subject Key Identifier" information for the target root certificate from below command
    

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

  • The "Subject Key Identifier" will be marked as "Authority Key Identifier" in the Machine_SSL certificate details. It can be fetched and verified by using below command. If you find it, which means that the vCenter is currently using the root certificate.
    

    /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text

Mandatory precaution:

• Ensure that all Platform Services Controllers in the federated environment are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems while powered off. 
• Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.