Aria Automation Patch fails with the error: Upgrade on VMware Aria Automation VA <FQDN> failed with state Unexpected Error

Products

• VMware Aria Suite

Issue/Introduction

• During a patch for Aria Automation, the workflow will fail at Stage 1 while collecting upgrade status from the Automation Appliance.
• During the patching Workflow it will move first to task Upgrade VMware Aria Automation and then will move back to Upgrade/Patch/Internal before failing with the common error: LCMVRAVACONFIG90030.  Example:

• If you execute the recommended query vracli upgrade status --details the response is: 

  • No upgrade is currently in progress.

• There are no identifiable errors or failures present in the Automation Logs.
• A review of the primary Automation node vami.log file will show that the proper version manifest was identified and downloaded but nothing happens after (note that the installed and downloaded versions may differ in your instance):

  • 11/09/2024 16:20:11 [INFO] New manifest. installed-version=8.16.2.34725, downloaded-version=8.16.2.34729

• A review of the vmware_vlcm.log from the Aria Suite Lifecycle Manager appliance shows the following:

  • 2024-09-11 17:08:12.263 ERROR [pool-3-thread-25] c.v.v.l.p.c.v.t.VraVaUpgradeStatusCheckTask -  -- Un-expected status seen after Upgrade on VMware Aria Automation VA <FQDN>. To know more about the failure, run command "vracli upgrade status --details" on the VMware Aria Automation VA <FQDN>. Upgrade is not successful, VMware Aria Automation appliance has been auto reverted to its previous working state. You can retry the upgrade by going to the product section.

Environment

VMware Aria Automation 8.x

Cause

Aria Suite Lifecycle Manager has sent the upgrade request but the Automation Appliances are not beginning the deployment.  Lifecycle Manager then polls Aria Automation for the following command:

vracli upgrade status | grep -A 6 "Summary" | grep "Description:" | cut -f 3

The response from Automation is No Upgrade Currently in Progress which is outside of the expected active or post-upgrade responses causing an "Unexpected Response" and failure.

Resolution

1. Snapshot or take a backup of all VMware Aria Automation Appliances.
2. Restart all appliances and wait for environment to come up healthy.  Validate by using the following command to ensure all pods are marked as running or completed:

   • kubectl get pods -n prelude

3. Execute the following command on each Automation node to ensure old upgrade and patch data has been cleaned up:

   • vracli cluster exec -- bash -c 'rm -rf /data/restorepoint /var/vmware/prelude/upgrade /var/log/vmware/prelude/upgrade-report-latest*; crontab -u root -l | grep -v -F "/opt/scripts/upgrade/upg-mon.sh" | crontab -u root -'

4. Retry the patch task from VMware Aria Suite Lifecycle Manager.