Upgrade fails with error: Operation Failed. If the error persists, contact Broadcom Support

Products

VMware Aria Suite

Issue/Introduction

Upgrade fails with message:

Operation Failed. If the error persists, contact Broadcom Support

Environment

vRealize Operations 8.x
Aria Operations 8.x

Cause

This may occur if the Ignore PAK file signature checkbox is checked on the Software Upgrade page.

You may find the following message in log /storage/log/vcops/log/casa/admin-ui.log:

2024-08-31T13:13:08,984+0000  INFO [ajp-nio-127.0.0.1-8011-exec-31] com.vmware.vcops.adminui.service.PakUploadService:91 - Upload solution name = vRealize_Operations_Manager_With_CP-8.10.x-to-8.16.0.23251565.pak, force = true, ignoreSignatureChecking = true

If ignoreSignatureChecking is 'true', this indicates that the checkbox were checked.

Issue may also occur if directory /usr/lib/vmware-casa-b2b is retained in /usr/lib, this signifies that there could have been an incomplete upgrade or APUAT installation. From /storage/log/vcops/log/casa/casa.log you may see the following message:

2024-10-02T12:06:47,162+0000  INFO [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] support.subprocess.GeneralCommand:255 - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-casa-b2b/casa-b2b/pakManager/bin/vcopsPakManager.py --action query_pak_history --json' threw exception: CommandLineExitException: key=general.failure; args=2,/usr/lib/vmware-python-3/bin/python: can't open file '/usr/lib/vmware-casa-b2b/casa-b2b/pakManager/bin/vcopsPakManager.py': [Errno 2] No such file or directory
; cause=
2024-10-02T12:06:47,162+0000  WARN [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] upgrade.pak.PakCommand:560 - Unable to parse resource result: com.vmware.vcops.casa.exception.CasaException: JSON error unmarshalling; class=com.vmware.vcops.casa.upgrade.pak.PakManDetail, json=
2024-10-02T12:06:47,163+0000  WARN [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] casa.exception.CasaControllerExceptionHandler:273 - general runtime exception
com.vmware.vcops.casa.exception.CasaException: Could not query_pak_history for pakID=null
        at com.vmware.vcops.casa.upgrade.pak.PakCommand.translateCommandLineException(PakCommand.java:561) ~[classes/:?]
        at com.vmware.vcops.casa.upgrade.pak.PakCommand.queryPakHistory(PakCommand.java:469) ~[classes/:?]
...

And from /storage/log/vcops/log/casa/admin-ui.log, you may see the following warning:

2024-10-02T12:06:47,165+0000  WARN [ajp-nio-127.0.0.1-8011-exec-6] com.vmware.vcops.adminui.action.SolutionAction:102 - NPE when processing latest solution status reponse from CASA
java.lang.NullPointerException: null
        at com.vmware.vcops.adminui.action.SolutionAction.getLatestInstalledSolutionStatuses(SolutionAction.java:86) ~[classes/:?]
        at com.vmware.vcops.adminui.action.SolutionAction.execute(SolutionAction.java:60) ~[classes/:?]
...

Resolution

Please try one step at a time from the list below, starting with step 1:

Ensure that you do not check the Ignore PAK file Signature checkbox when performing an upgrade for the appliance. This checkbox is not applicable for the appliance upgrade PAKs.
If step 1 do not resolve, it's recommended to bring the cluster offline from Admin UI and then restart the Operations nodes from the vCenter as per KB 340318.
This is to ensure that all services has a clean start, and avoid potential issues were CASA service do not fully stop, or has a second running zombie process.
If step 1 and 2 does not resolve, verify if directory /usr/lib/vmware-casa-b2b is present. This directory only exists during an upgrade process or APUAT execution, and may have been left on the system after an incomplete upgrade. To resolve, remove the usr/lib/vmware-casa-b2b directory and restart casa service as per steps below. Make sure you have snapshots before following this step!

Log in to node(s) with console or SSH using root account

cd /usr/lib
rm -Rf vmware-casa-b2b

## Alternatively, move this directory somewhere else, like /tmp
mv /usr/lib/vmware-casa-b2b /tmp

Restart CASA service with:
```
service vmware-casa restart
```
Retry running upgrade or APUAT