vCenter Upgrade Precheck Fails Due to Expired BACKUP_STORE Certificates

Products

• VMware vCenter Server

Issue/Introduction

• When performing pre-upgrade checks on vCenter Server 7.x or higher in preparation for an upgrade to a vCenter, the certificate status report from the vCert - Scripted vCenter Expired Certificate Replacement tool  indicates an expired entry in the BACKUP_STORE.
  
  
• The following status is observed in the certificate report:

Checking Certificate Status
-----------------------------------------------------------------
Checking Machine SSL certificate                            VALID
Checking Solution User certificates:
   machine                                                                VALID
   vsphere-webclient                                                 VALID
   vpxd                                                                      VALID
   vpxd-extension                                                     VALID
   hvc                                                                        VALID
   wcp                                                                       VALID
Checking SMS self-signed certificate                      VALID
Checking data-encipherment certificate                  VALID
Checking Authentication Proxy certificate               VALID
Checking Auto Deploy CA certificate                      NO SKID
Checking VMDir certificate                                     VALID
Checking BACKUP_STORE entries:
   bkp___MACHINE_CERT                                   EXPIRED
   bkp_machine                                                     VALID
   bkp_vsphere-webclient                                      VALID
   bkp_vpxd                                                           VALID
   bkp_vpxd-extension                                          VALID
Checking legacy Lookup Service certificate         VALID
Checking VMCA certificate                                   VALID

 

Environment

VMware vCenter Server

Cause

• When replacing certificates in vCenter Server Appliance, 2 backup stores are being created, BACKUP_STORE and BACKUP_STORE_H5C. During the replacement, the old certificates are added as entries in these stores to allow for a rollback.
• When any of the entries in this store is expired it will show an expired entry in the BACKUP_STORE section of the report. 

Resolution

Before you begin: 

• Ensure an up-to-date offline snapshot of the VCSA has been taken. 
• For Enhanced Linked Mode (ELM), take snapshots of all linked vCenter nodes simultaneously while powered off.

 

Step-by-Step Instructions:

1. Upload vCert Utility:

   • Use WinSCP to upload the vCert.zip file to the /tmp directory of the VCSA.

   • Note: If WinSCP fails with a "Received too large SFTP packet" error, modify the WinSCP connection settings:

     • Open Advanced Settings > SFTP.

     • Set SFTP server to: shell /usr/libexec/sftp-server.

2. Install and Launch vCert:

   • SSH to the vCenter as root and navigate to the upload directory: cd /tmp.

   • Unzip the tool: unzip vCert-x.x.x.zip.

   • Navigate to the directory and run the script: ./vCert.py

3. Clear Expired Certificates:

   • In the vCert main menu, select Option 3: Manage Certificates.

   • From the sub-menu, select Option 12: Clear expired certificates in BACKUP_STORE in VECS.

4. Re-run the vCenter upgrade precheck to confirm the certificates are now valid or removed.

   • In the vCert main menu, select 1: Check current certificate status
   • The output should now show valid for all certificates. 

Additional Information

Post replacing the vCenter's machine SSL & solution users' certificates as well, it's possible that the BACKUP_STORE would still show old certificates (Validity X days)

Where X is the number of days the old certs would have expired.

Once, it is confirmed that the certificates are successfully replace, then the above resolution steps could be followed to remove the old certificates.