Upgrade fails with error: Operation Failed. If the error persists, contact Broadcom Support

Products

VMware Aria Suite

Issue/Introduction

Upgrade fails with message:

Operation Failed. If the error persists, contact Broadcom Support

Environment

vRealize Operations 8.x
Aria Operations 8.x

Cause

This may occur if the Ignore PAK file signature checkbox is checked on the Software Upgrade page.

You may find the following message in log /storage/log/vcops/log/casa/admin-ui.log:

2024-08-31T13:13:08,984+0000  INFO [ajp-nio-127.0.0.1-8011-exec-31] com.vmware.vcops.adminui.service.PakUploadService:91 - Upload solution name = vRealize_Operations_Manager_With_CP-8.10.x-to-8.16.0.23251565.pak, force = true, ignoreSignatureChecking = true

If ignoreSignatureChecking is 'true', this indicates that the checkbox was checked.

The issue may also occur if the directory /usr/lib/vmware-casa-b2b is retained in /usr/lib. This signifies that there could have been an incomplete upgrade or APUAT installation. From /storage/log/vcops/log/casa/casa.log you may see the following message:

2024-10-02T12:06:47,162+0000  INFO [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] support.subprocess.GeneralCommand:255 - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-casa-b2b/casa-b2b/pakManager/bin/vcopsPakManager.py --action query_pak_history --json' threw exception: CommandLineExitException: key=general.failure; args=2,/usr/lib/vmware-python-3/bin/python: can't open file '/usr/lib/vmware-casa-b2b/casa-b2b/pakManager/bin/vcopsPakManager.py': [Errno 2] No such file or directory
; cause=
2024-10-02T12:06:47,162+0000  WARN [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] upgrade.pak.PakCommand:560 - Unable to parse resource result: com.vmware.vcops.casa.exception.CasaException: JSON error unmarshalling; class=com.vmware.vcops.casa.upgrade.pak.PakManDetail, json=
2024-10-02T12:06:47,163+0000  WARN [ajp-nio-127.0.0.1-8011-exec-10] [M90002u0] casa.exception.CasaControllerExceptionHandler:273 - general runtime exception
com.vmware.vcops.casa.exception.CasaException: Could not query_pak_history for pakID=null
        at com.vmware.vcops.casa.upgrade.pak.PakCommand.translateCommandLineException(PakCommand.java:561) ~[classes/:?]
        at com.vmware.vcops.casa.upgrade.pak.PakCommand.queryPakHistory(PakCommand.java:469) ~[classes/:?]
...

And from /storage/log/vcops/log/casa/admin-ui.log, you may see the following warning:

2024-10-02T12:06:47,165+0000  WARN [ajp-nio-127.0.0.1-8011-exec-6] com.vmware.vcops.adminui.action.SolutionAction:102 - NPE when processing latest solution status reponse from CASA
java.lang.NullPointerException: null
        at com.vmware.vcops.adminui.action.SolutionAction.getLatestInstalledSolutionStatuses(SolutionAction.java:86) ~[classes/:?]
        at com.vmware.vcops.adminui.action.SolutionAction.execute(SolutionAction.java:60) ~[classes/:?]
...

This issue can also occur if the /storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json contains corrupted entries. An example of this is the vcopsPakManagerCommonHistory.json contained the entry ]me_updated":"1726476788137" }, . This did not match the syntax of other entries in the file. You may see the following error in /storage/log/vcops/log/casa/casa.log

2025-04-25T07:26:53,940+0000  INFO [ajp-nio-127.0.0.1-8011-exec-8] [iL00003o] support.subprocess.GeneralCommand:255 - Command '/usr/bin/sudo -n /usr/lib/vmware-python-3/bin/python /usr/lib/vmware-vcopssuite/utilities/pakManager/bin/vcopsPakManager.py --action deploy_new_upgrade_content --pak vRealizeOperationsManagerEnterprise-818324663027 --json --force_content_update false --roles ADMIN,DATA,UI' threw exception: CommandLineExitException: key=general.failure; args=1,; cause=
2025-04-25T07:26:53,940+0000  WARN [ajp-nio-127.0.0.1-8011-exec-8] [iL00003o] upgrade.pak.PakCommand:590 - Unable to parse resource result: com.vmware.vcops.casa.exception.CasaException: JSON error unmarshalling; class=com.vmware.vcops.casa.upgrade.pak.PakManDetail, json=2025-04-25T07:26:53,905+0000 [3482] - root - An unhandled exception occurred, exiting with exit code: 1,
Type: "<class 'AttributeError'>"

Resolution

Please try one step at a time from the list below, starting with step 1:

Ensure that you do not check the Ignore PAK file Signature checkbox when performing an upgrade for the appliance. This checkbox is not applicable for the appliance upgrade PAKs.
If step 1 do not resolve, it's recommended to bring the cluster offline from Admin UI and then restart the Operations nodes from the vCenter as per KB 340318.
This is to ensure that all services has a clean start, and avoid potential issues were CASA service do not fully stop, or has a second running zombie process.
If step 1 and 2 does not resolve, verify if directory /usr/lib/vmware-casa-b2b is present. This directory only exists during an upgrade process or APUAT execution, and may have been left on the system after an incomplete upgrade. To resolve, remove the usr/lib/vmware-casa-b2b directory and restart casa service as per steps below. Make sure you have snapshots before following this step!

Log in to node(s) with console or SSH using root account

cd /usr/lib
rm -Rf vmware-casa-b2b

## Alternatively, move this directory somewhere else, like /tmp
mv /usr/lib/vmware-casa-b2b /tmp

Restart CASA service with:
```
service vmware-casa restart
```
Retry running upgrade or APUAT

If you still have an issue after trying the above steps please check for corruption in /storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json file.

Edit the vcopsPakManagerCommonHistory.json and remove corrupted entries

Log in to node(s) with console or SSH using root account
Edit the file using the command vi /storage/db/pakRepoLocal/vcopsPakManagerCommonHistory.json
Delete unwanted entries.
Make sure the file contains a "[" as a first non-space character and "]" as the last non-space character, If not, add the character to the file as shown in the screenshot below:
Save the file: wq!
Validate the same in all nodes.
Retry running upgrade