Installing a PAK file in vRealize Operations Manager results in the error "upgrade.verification.certificate_expired"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Installing an upgrade PAK, or Solution into vRealize Operations Manager fails with the error:

upgrade.verification.certificate_expired

Environment

VMware vRealize Operations Manager 7.0.x
VMware vRealize Operations Manager 7.5.x
VMware vRealize Operations Manager 6.7.x

Cause

An expired certificate is stuck in the Keystore or Truststore.
These certificates are typically left over from older Management Packs.

Resolution

To resolve this issue, delete the expired certificate from the Keystore and/or the Truststore.
Complete all steps, as only one store may contain expired certificates.

1. Log into the vRealize Operations Manager Primary node as root via SSH or Console.
2. View storePass.properties and make note of the Truststore and Keystore passwords:

cat /storage/vcops/user/conf/ssl/storePass.properties

Example:
sslkeystorePassword=N1bk+kOFmflMXfWpkK+y2eTVr26z6
ssltruststorePassword=p4NOg/+otdDL2bvag7FdUEETp+rPea

3. List the certificates in the Keystore:

keytool -list -v -keystore /data/vcops/user/conf/ssl/tcserver.keystore

Note: Enter the Keystore password noted in step 2 when prompted.

4. In the list of certificates, note the Alias name of any with an expired until date.

Example: A complete certificate entry will look as follows, with the Alias name and until date in bold:
Alias name: deep_security_manager
Creation date: Feb 11, 2016
Entry type: trustedCertEntry

Owner: CN=vxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Issuer: CN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Serial number: 520f0e5500000000562a
Valid from: Thu Jul 10 18:55:26 CEST 2014 until: Sat Jul 09 18:55:26 CEST 2016
Certificate fingerprints:
         MD5:  
         SHA1: D2:D4:19:46:28:A8:AB:FB:4E:10:50:7D:15:51:20:F2:AA:D3:F4:E7
         SHA256: 01:0A:43:7D:87:0B:E9:22:FA:32:C3:5A:C3:5C:D5:B0:0B:CE:40:56:E3:14:3E:A7:9E:6A:8B:C1:DD:7B:EA:E1
Signature algorithm name: SHA1withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

5. Delete any expired certificates from the Keystore:

keytool -delete -alias alias_name -keystore /data/vcops/user/conf/ssl/tcserver.keystore -storepass keystore_password

Note: Replace alias_name with the Alias name noted in step 4, and replace keystore_password with the Keystore password noted in step 2.

6. List the certificates in the Truststore:

keytool -list -v -keystore /data/vcops/user/conf/ssl/tcserver.truststore
 

Note: Enter the Truststore password noted in step 2 when prompted.

7. In the list of certificates, note the Alias name of any with an expired until date.
8. Delete any expired certificates from the Keystore:

keytool -delete -alias alias_name -keystore /data/vcops/user/conf/ssl/tcserver.truststore -storepass truststore_password

Note: Replace alias_name with the Alias name noted in step 6, and replace truststore_password with the Keystore password noted in step 2

9. Attempt to install the upgrade PAK, or Solution again.

Remove the previously made snapshots once everything is confirmed to be working.

Additional Information

How to take a Snapshot of vRealize Operations Manager 6.x and later
Removing a solution from vRealize Operations Manager 6.x and later

Impact/Risks:

If a needed certificate is deleted, it can cause issue with vRealize Operations Manager itself, or collections.
Take a snapshot of the vRealize Operations Manager nodes before proceeding with the Resolution section.

If an old certificate is determined to be in use by an installed Management Pack, see Removing a solution from vRealize Operations Manager 6.x and later to remove the Management Pack.
Alternatively, you can try upgrading the Management Pack if an upgrade is available.