Certificate Status Alarm - Clearing BACKUP_STORES certificates in the VCSA

Products

• VMware vCenter Server

Issue/Introduction

• Following alarm is present on vCenter Object in vSphere Client

         
          

• vCenter Certificates have been confirmed to be valid or been regenerated recently
  
  
• Reviewing the certificate stores shows expired certificates within BACKUP_STORE or BACKUP_STORE_H5C in VECS 
  • Run the below command to see the status of the environmental certificates:

                  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

 

 

Environment

• vCenter 6.x
• vCenter 7.x
• vCenter 8.x

Cause

• When replacing certificates in vCenter Server Appliance, 2 backup stores are being created, BACKUP_STORE and BACKUP_STORE_H5C.
• During the replacement, the old certificates are added as entries in these stores to allow for a rollback.
• When any of the entries in these store is expired or about to expire, vSphere Client will display a Certificate Status Alarm.

Resolution

The script previously attached to this KB is deprecated.

Option 1:

Use the new improved certificate management tool vCert - Scripted vCenter Expired Certificate Replacement for all certificate management/replacement workflow. 

1. Download and install vCert on the vCenter Server Appliance as described in Installation Section.
2. Use Option 12 - Clear expired certificates in BACKUP_STORE in VECS from menu  Manage Certificates to clear the expired certificates from the backup stores.

Note: Use WinSCP to upload the vCert tool to VCSA. For additional information, see Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B.

The alarm has been fixed in 7.0 U3 and 8.0 so that an expired BACKUP_STORE certificate will not trigger an alarm.
However, the BACKUP_STORE_H5C will trigger the alarm, so certificates in BACKUP_STORE_H5C must be deleted to suppress the alarm.

 

If vCenter fails to clear the BACKUP_STORE, try the manual method as given below

 

Option 2:

1. Take a snapshot of the VCSA and, if in Linked Mode, all PSCs while powered off.
2. SSH into vCenter as root and list certificates:
3. Run the below command to list the store

        /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE

     4. Make a note of the alias of the certificate that you want to remove from the BACKUP_STORE

     5. Run the below command to remove the expired certificate:

        /usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store BACKUP_STORE --alias <cert_alias>

Additional Information

Mandatory precaution:

• Ensure that the required offline snapshot of VCSA is taken.

When using Enhanced Linked mode(ELM):

• Ensure that all Platform Services Controllers in the federated environment (ELM, enhanced linked mode) are shut down and take a snapshot of all of them. They should be powered down to ensure that no replication takes place partially during the snapshot operation. Power On all the PSCs when the snapshot operation is complete. Also, take snapshots of the vCenter Systems. 
• Snapshot revert (If required to recover from a damage) should happen on all the nodes to the same powered off snapshot state to ensure replication data consistency.
• This script should only be run on External PSC and VCSA's that have BACKUP_STORES expired or expiring.