Aria Automation upgrade from 8.16.2 to 8.17 via Aria Lifecycle fails after 4 hours with Error Code: LCMVRAVACONFIG90030

Products

• VMware Aria Suite

Issue/Introduction

After triggering the upgrade from Arai Lifecycle, the upgrade fails after 4 hours with Error Code: LCMVRAVACONFIG90030

"Welcome to VMware Aria Automation Appliance 8.17.0" shows in the upgrade-*.log so the appliances have been upgraded to 8.17, and the upgrade is waiting on the appliances to be rebooted

 

Environment

Aria Lifecycle 8.16 PSPack 3

Aria Automation 3 node cluster

Cause

Either reboot did not occur post upgrade, or the services did not start automatically after the reboot, as the pods kube-system and prelude namespaces were not running. The console landing pages also show that the appliance are at 8.17.
The Aria Lifecycle upgrade request will timeout and fail if the Services are not up and running after 4 hours. 

Resolution

1. Take non-memory snapshots of Aria Automation nodes. 

2. Reboot the appliances manually. 
3. Once the landing page in the console shows, log in and run /opt/scripts/deploy.sh on one node to start the services.
4. After all pods are running, clean up the upgrade progress but running this script on one node:

vracli cluster exec -- bash -c 'rm -rf /data/restorepoint /var/vmware/prelude/upgrade /var/log/vmware/prelude/upgrade-report-latest*; crontab -u root -l | grep -v -F "/opt/scripts/upgrade/upg-mon.sh" | crontab -u root -'