Certificate alarm - Clearing BACKUP_STORES certificates in the VCSA

Products

• VMware vCenter Server

Issue/Introduction

Help automating the clean-up of expired Certificates from the BACKUP_STORES and BACKUP_STORE_H5C from the VCenter Server Appliance.

Symptoms:
Despite the vCenter certificates already having been confirmed as valid, or having been regenerated, vSphere Client still continues to display certificate status alarms, when there are any expired certificates contained in either BACKUP_STORE or BACKUP_STORE_H5C

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

When replacing certificates in VCSA , 2 backup stores are being created, BACKUP_STORE and BACKUP_STORE_H5C. During the replacement, the old certificates are added as entries in these stores to allow for a rollback.
When any of the entries in these store is expired or about to expire, vSphere Client will display a Certificate Status Alarm.

Resolution

To resolve the issue follow the steps below:

1. Ensure that the required snapshot or offline snapshots have been taken as recommended in the "Impact" section.
2. Check the certificates with the following command: 

   # for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

3. If there are expired or Expiring Certificates under STORE BACKUP_STORE or STORE BACKUP_STORE_H5C, please proceed by running the attached Script clean_backup_stores.sh
4. Upload the attached script to the VCSA or external PSC to the /tmp/ directory.
5. Make the script executable by running:

   # chmod +x clean_backup_stores.sh

6. Run the script with the command:

   ./clean_backup_stores.sh

7. The script will first verify if there are indeed any expired certificates in one of the backup stores (BACKUP_STORE and BACKUP_STORE_H5C). If it finds any, it will ask to confirm that powered off snapshots have been taken of the VCSA and of all of it's replication partners:

   Have you taken powered off snapshots of all PSC's and VCSA's within the SSO domain(Y|y|N|n)

8. Confirm the question with a "Y" or "y" if you have following the recommendation outlined in the "Impact" section of this article. Otherwise ensure to create the necessary snapshot or offline snapshots now, then run the script again.
9. In some scenarios, the script can fail with an error: 

   -bash: ./clean_backup_stores.sh: /bin/bash^M: bad interpreter: No such file or directory.

10. Should you see this error, run the following command:

    # sed -i -e 's/\r$//' clean_backup_stores.sh

11. Restart the vCenter services with the command:

    # service-control --stop --all && service-control --start --all

12. Proceed to "Acknowledge" and "Reset to green" any Certificate related alarm.
13. If there are expired or expiring certificates in the Trusted Roots store please refer to KB Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store (VECS) (2146011)

Note: You may use WinSCP to upload the script to VCSA. For additional information, see Error when uploading files to vCenter Server Appliance using WinSCP (2107727).
 

Additional Information

Impact/Risks:

Mandatory precaution:

• If you are working on a single VCenter with external or embedded PSC the snapshots could be taken while powered On.

When using Enhanced Linked mode(ELM):

• Ensure that all Platform Services Controllers in the federated environment (ELM, enhanced linked mode) are shut down and take a snapshot of all of them. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems. 
• Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.
• This script should only be run on External PSC and VCSA's that have BACKUP_STORES expired or expiring.