Upgrading VMware Aria Operations to 8.14 fails with error "resource key=pak_manager.action_failed, resource args=[run admin first boot scripts]"

Products

• VMware Aria Suite

Issue/Introduction

Upgrading VMware Aria Operations to 8.14 fails at step 9 of 14 with the following error message: "resource key=pak_manager.action_failed, resource args=[run admin first boot scripts]"

/storage/log/vcops/logs/pakManager/vRealizeOperationsManagerEnterprise-814022611353/apply_system_update_stderr.log:

2023-11-09T02:33:29 ERROR [6145] - root - [Failed] /usr/lib/vmware-casa/firstboot/casa-unicorn-firstboot.py casa-unicorn-firstboot.py - Failed
2023-11-09T02:33:29 INFO [6145] - root - Done with all scripts
2023-11-09T02:33:29 INFO [6145] - root - Result:
2023-11-09T02:33:29 INFO [6145] - root - --------------------------
2023-11-09T02:33:29 INFO [6145] - root - notRun scripts:
2023-11-09T02:33:29 INFO [6145] - root - Run scripts:
2023-11-09T02:33:29 INFO [6145] - root - [Finished] /usr/lib/vmware-vcopssuite/firstboot/vsutilitiesFirstBoot.py
2023-11-09T02:33:29 INFO [6145] - root - [Finished] /usr/lib/vmware-casa/firstboot/casa-firstboot.py
2023-11-09T02:33:29 INFO [6145] - root - [Finished] /usr/lib/vmware-casa/firstboot/admin-app-firstboot.py
2023-11-09T02:33:29 INFO [6145] - root - [Failed] /usr/lib/vmware-casa/firstboot/casa-unicorn-firstboot.py
2023-11-09T02:33:29 ERROR [6145] - root - Upgrade first boot flow is a failure

/var/log/firstboot/casa-unicorn-firstboot.py_6145_stdout.log:

2023-11-09T02:33:29,405+0000 [7887] - root - Entering vcops-web-admin firstboot 2023-11-09 12:33:29
2023-11-09T02:33:29,405+0000 [7887] - root - Deploying war ...
2023-11-09T02:33:29,405+0000 [7887] - root - web source file vrops-casa-unicorn
2023-11-09T02:33:29,405+0000 [7887] - root - Checking if the /usr/lib/vmware-casa/casa-webapp/webapps dir exists
2023-11-09T02:33:29,405+0000 [7887] - root - Dir: /usr/lib/vmware-casa/casa-webapp/webapps does exist
2023-11-09T02:33:29,405+0000 [7887] - root - Source file war: "/usr/lib/vmware-casa/vrops-casa-unicorn.war" does not exist

If you look at the /opt/vmware/etc/ovfEnv.xml this instance has been deployed via vCenter adapter- Solution install:

<Property oe:key="solutionInstall.postDeployData.endpoint" oe:value="https://<vcenter.marvel.com>/api/ui/solutioninstall"/>
<Property oe:key="solutionInstall.postDeployData.thumbprint"oe:value="thumbprint value"/>
<Property oe:key="solutionInstall.token" oe:value="value"/> 

Environment

VMware Aria Operations 8.14.x

Cause

This is a known issue related to vCenter plugin integration into Aria operations. It leads to product upgrade failure if the cluster has been deployed through the vCenter plugin window. Unicorn's 'casa-unicorn-firstboot.py' script shouldn't be executed during the product update. The vCenter plugin webapp installation and configuration must only be performed at fresh deployment.

Resolution

The issue is fixed in 8.14 hot fix 1 KB-324354

Workaround:

Note: This has to be performed on all the Analytics nodes in the cluster

1. Revert to the snapshot that was taken prior to the upgrade. 

2. Take a SSH to the node and backup the /etc/vmware/etc/ovfEnv.xml file

#cp /etc/vmware/etc/ovfEnv.xml /etc/vmware/etc/ovfEnv.xml.bkp

3. Take the cluster offline and Take a snapshot of the vROps cluster

From the vCenter Server Web interface, perform a guest shutdown on the Aria operations VM.
From the vCenter Server Web interface, right-click the Aria operations VM and click Configure.
Navigate to vApp Options > Properties.
Select the below property and delete it

solutionInstall.postDeployData.endpoint
solutionInstall.postDeployData.thumbprint
solutionInstall.token

Note: Please note the property and value so you can add this back after the upgrade.

4. Start the upgrade

5. Once the upgrade is successful. Take the cluster offline

6. Power off the nodes and add the property back in vApp Options

 

 

Additional Information

Impact/Risks:

Upgrade failure