Aria Operations 8.x upgrade fails: "Signature on PAK file is not valid"

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:
At the time of failure, the following will be present:

• The VMware Aria Operations Admin UI will display a message similar to:

Signature on PAK file 'vRealize_Operations_Manager-VA-8.x-to-8.2.0.18707701.pak' is not valid.

Note: Your PAK file name may differ.

• The /storage/log/vcops/log/casa/casa.log file shows entries similar to

  INFO [ajp-nio-127.0.0.1-8011-exec-86] [AP003Akt] support.subprocess.GeneralCommand:255 - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py --action query_pak_signature --pak /storage/db/casa/pak/dist_pak_files/cargo/reservedTemp_8158315163481856854.pak --json' threw exception: CommandLine ExitException: key=general.failure; args=1,; cause=

• The /storage/log/vcops/log/pakManager/vcopsPakManager.root.query.log file shows entries similar to

  INFO [20752] - vcopsPakSignatureValidation.getExpectedCertificates:335 - Unable to retrieve Software Publishing Certificate with alias vmwarespc from truststore /storage/vcops/user/conf/ssl/tcserver.truststore.

Environment

VMware Aria Operations 8.x
VMware vRealize Operations 8.x

Cause

This issue is caused by a vmwarespc certificate missing from the truststore of VMware Aria Operations (formerly known as VMware Aria Operations).

Resolution

To resolve this issue, install the missing vmwarespc certificate in the truststore.

• Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
• Run the following command to install the vmwarespc certificate in the truststore

  /usr/java/latest/bin/keytool -importcert -trustcacerts -alias vmwarespc -file /usr/lib/vmware-vcopssuite/utilities/conf/vmwarespc.pem -keystore /storage/vcops/user/conf/ssl/tcserver.truststore -storepass "$(sed -n 's/^ssltruststorePassword=//p' /data/vcops/user/conf/ssl/storePass.properties)" -noprompt

• Re-attempt the upgrade in the VMware Aria Operations Admin UI.
  
  

• If the above doesn't help and  you see below entries in the /storage/log/vcops/log/casa/casa.log: 

  2025-02-05T04:54:32,666+0000  INFO [ajp-nio-127.0.0.x.x8011-exec-19] [oQ0000wa] casa.support.RequestIdIncomingInterceptor:68 - Request GET /casa/upgrade/slice/private/webapp/restarting

  from 127.0.0.1: New request id oQ0000wa
2025-02-05T04:54:32,666+0000  INFO [ajp-nio-127.0.0.x.-80xx-exec-19] [oQ0000wa] casa.support.RequestIdIncomingInterceptor:105 - Request GET /casa/upgrade/slice/private/webapp/restarting
: Done
2025-02-05T04:54:32,734+0000 ERROR [ajp-nio-127.0.0.x.x.-8011-exec-16] [oQ0000wL] upgrade.pak.PakService:4913 -
com.vmware.vcops.casa.exception.CasaLocalizableException: CasaLocalizableException: key=pak_manager.pakfile.invalid_signature; args=vRealize_Operations_Manager_With_CP-8.14.x-to-8.18.3
.24521385 (1).pak,The signature manifest file contains an invalid hash: "SHA1(proxy-remote-collector/vRealize-Operations-Cloud-Proxy-8.18.3.24521398.pak)= 3064f39b9071767e105a3250bb0d9
5082926b221"; cause=
at com.vmware.vcops.casa.upgrade.pak.PakService.uploadLocal(PakService.java:489) ~[classes/:?]
        at com.vmware.vcops.casa.upgrade.pak.PakService.upload(PakService.java:358) ~[classes/:?]
        at com.vmware.vcops.casa.upgrade.pak.PakService.uploadToMaster(PakService.java:4904) ~[classes/:?]
        at 

   

  • Remove the existing alias from the truststore:  /usr/java/latest/bin/keytool -delete -alias vmwarespc -keystore /storage/vcops/user/conf/ssl/tcserver.truststore -storepass "$(sed -n 's/^ssltruststorePassword=//p' /data/vcops/user/conf/ssl/storePass.properties)"

   

  •  Re-import the Certificate: After deleting the existing alias, re-import the certificate:
    /usr/java/latest/bin/keytool -importcert -trustcacerts -alias vmwarespc -file /usr/lib/vmware-vcopssuite/utilities/conf/vmwarespc.pem -keystore /storage/vcops/user/conf/ssl/tcserver.truststore -storepass "$(sed -n 's/^ssltruststorePassword=//p' /data/vcops/user/conf/ssl/storePass.properties)" -noprompt
    Retry the Upgrade: Once the certificate is re-imported, attempt the upgrade again.