Remove Expired and Unused CA certificates from Trusted_ROOTs store in the VMware Endpoint Certificate Store(VECS) via Script.
search cancel

Remove Expired and Unused CA certificates from Trusted_ROOTs store in the VMware Endpoint Certificate Store(VECS) via Script.

book

Article ID: 309888

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Clean Trusted Roots Certificate Store from VCSA in a scripted way.
  • You see a critical alarm in the vSphere Client or vSphere Web Client for a Certificate expiry.
  • A CA Certificate that is in use in the environment is expiring or expired.
  • You have already renewed the certificates and have a new, valid CA Certificate in place. Remove expired old SSL certificate.
  • Attempts to remove the expired CA Certificate using the Web Client or other methods fail, and the Certificate is copied back to VMware Endpoint Certificate Store (VECS) after deletion.
  • Remove/delete trusted root certificate.

Environment

  • VMware vCenter Server 8.0.x
  • VMware vCenter Server 7.0.x
  • VMware vCenter Server 6.7.x
  • VMware vCenter Server 6.5.x

Cause

Certificates are copied back to the VECS store because the CA Certificate which is expiring is published to the VMware Directory Service (VMDIR). When the Certificate is removed from VECS, VMDIR adds the Certificate back to VECS during a sync operation. This is done in order to ensure the integrity of the TRUSTED_ROOTS Certificate store, as deletion of an incorrect Certificate from this store could cause the environment to be irreparably damaged.

Resolution

To un-publish expired/expiring/unwanted certificates from TRUSTED_ROOTS VECS Store:

  1. Take snapshot of VCSA VM (If VCs are in ELM then take powered off snapshots of all VCs in ELM)
  2. Download the clean_trusted_roots.sh script attached to this article
  3. Upload the script to the VCSA with embedded PSC or external PSC in the /tmp folder or copy its contents to a text file on the appliance using vi.

    Note: You may use WinSCP to upload the script to VCSA. For additional information, see Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B.

  4. SSH to VCSA with root
  5. Browse to /tmp directory 
    cd /tmp
  6. Make the file executable 
    chmod +x clean_trusted_roots.sh
  7. Execute the script 
    ./clean_trusted_roots.sh


    Sample output

    root@vc [ ~ ]# ./clean_trusted_roots.sh

This Script validates and allows to clear Trusted Roots Certificate Stores from SSO Domain.

Proceed with EXTREME CAUTION. If the wrong Certificate is un-published and removed from VECS, this can damage the environment which can be irreparable. This may happen if the replication between PSC's is not working.
Be absolutely certain that the Certificate you are removing is the correct Certificate to remove.
Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.

Mandatory precaution:

Ensure that all Platform Services Controllers in the federated environment are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems while powered off.

Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.

#######

Please enter the administrator@vsphere. local password:

     

    Note:  You may receive an error when you try to run the script:

    bash:  ./clean_trusted_roots.sh: /bin/bash^M: bad interpreter: No such file or directory


    This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.  To resolve this problem, run the following command and rerun the script:

    sed -i -e 's/\r$//' clean_trusted_roots.sh
  8. Enter the SSO administrator password when prompted.

    Note: If wrong credentials are provided or if certificates from another PSC/VCSA are not readable you will see the following error 
    Removable not in use Certificates in SSO domain are:
There are no removable Certificate CN ID's from Trusted Roots Store.
There was an error reading the Certificates, please make sure your entered correct SSO admin password and all services are running in all VCSA's in SSO and check replication between PSC's


    If credentials are correct the script will execute, and you should see the below output

    Sample output

    Removable not in use Certificates in SSO domain are:
154AC6461E82344D1A03E5B72FDA9626F5C03912

CN ID's file is not empty, continuing ...

Do you want to proceed removing the Removable CN ID's (Y|y|N|n)yCertificate retrieved successfully

current CN ID's in use after clean up are:
Number of certificates: 2
#1:
CN( id):
Subject DN:
CRL present:
#2:
CN(id):
Subject DN:
CRL present:

242B83C799A98461844002432D0AE9F1EE87C6AF
CN=vc2.example.com, DC=vsphere, DC=local, C=US, ST=California, 0=vc2.example.com, OU=VMware Engineering
yes

AF681738D9368F787D28922D7402CC917FC049FA
CN=vc1.example.com, DC=vsphere, DC=local, C=US, ST=California, 0=vc1.example.com, OU=VMware Engineering
yes
  9. Restart all services on the PSCs /VCSAs and ensure that all services start and respond normally and that you can log in and manage the environment. 
    service-control --stop --all && service-control --start--all



Manual method to perform these steps are defined in follow KB Removing CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS)

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ

Refer to KB CertificateStatusAlarm - There are certificates that expired or about to expire / Certificate Status Change Alarm Triggered on VMware vCenter Server for more information on removing expired certificates from other certificate stores.

Impact/Risks:

  • WARNING:
    • Proceed with EXTREME CAUTION. If the wrong Certificate is un-published and removed from VECS, this can damage the environment which can be irreparable.
    • Be absolutely certain that the Certificate you are removing is the correct Certificate to remove.
    • Validate the root certificate which is about to expire is renewed and all certificates from this root certificate are also renewed/replaced before un-publishing.
  • Mandatory precaution:
    • Ensure that all Platform Services Controllers in the federated environment are shut down and take a snapshot of all of them while they are powered off. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems while powered off.
    • Snapshot revert (If required to recover from damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.
    • Please make sure the Replication between PSC's in the Whole SSO is accurate prior to running this script. Failure to do so may lead to the Removal of an incorrect Certificate or leave leftovers.



Attachments

clean_trusted_roots.sh get_app
Powered by Wolken Software