Troubleshooting VMware Aria Suite Lifecycle 8.x upgrade failures due to missing packages

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
The VMware Aria Suite Lifecycle 8.x upgrade failure can be caused due to many other symptoms as well but for this KB the relevant ones are as follows

You trigger VMware Aria Suite Lifecycle 8.x upgrade and after running through a few of the initial steps, the upgrade progress in UI seems to be frozen. If left alone, the UI remains at this stage for hours.
Similar errors to the below are found in the /opt/vmware/var/log/vami/vami.log

12/09/2022 09:39:20 [INFO] Using update pre-install script
12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing

Environment

VMware Aria Suite Lifecycle 8.x
VMware vRealize Suite Lifecycle Manager 8.x

Cause

The VMware Aria Suite Lifecycle 8.x upgrade process fails to capture the required binary files from the upgrade source. Both online and CD-ROM upgrade source methods are impacted.

Resolution

See the Workaround section below for additional information.

Workaround:

Prerequisites

Please take simultaneous non-memory snapshots of the virtual appliance(s).
You have access to root user and password.
You have SSH or console access to the virtual appliance.
Download the upgrade ISO from VMware Customer Connect.

Troubleshooting Procedure

Revert the appliance to its pre-upgrade snapshot.
Enable the debug mode for the vami log by running the following command
```
sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml
```
Tail the vami.log using the command below, then re-run the upgrade (like it was done initially when the failure first occurred)
```
tail -f /opt/vmware/var/log/vami/vami.log
```

In debug mode, the missing packages will be listed. Record this information for use in later steps. See below for an example of a missing package

14/09/2022 10:42:34 [DEBUG] Stat file, file missing. filename=/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/audit-2.8.5-17.ph3.x86_64.rpm

Workaround

As a first step check for the existence of a stale upgrade status file.

vami.log will show below

27/10/2022 11:17:15 [DEBUG] Lock file 1
27/10/2022 11:17:15 [DEBUG] status file /opt/vmware/var/lib/vami/update/data/job/2/status
27/10/2022 11:17:15 [DEBUG] cli failed
27/10/2022 11:17:15 [DEBUG] cache path is/opt/vmware/var/lib/vami/update/data
27/10/2022 11:17:15 [DEBUG] No next version exists
27/10/2022 11:17:15 [DEBUG] No next version exists
27/10/2022 11:17:15 [DEBUG] Setting download cache location: /opt/vmware/var/lib/vami/update/data
27/10/2022 11:17:15 [DEBUG] Disabling old download cache location:
27/10/2022 11:17:15 [DEBUG] Done setting download cache location
27/10/2022 11:17:15 [DEBUG] skipClearDownloadCache =
27/10/2022 11:17:15 [DEBUG] Clear Download Cache. path=/opt/vmware/var/lib/vami/update/data

and upgrade again fails with below error:

12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing

Take the erroneous job ID from above snippet and go to that folder and check contents of status file.

less /opt/vmware/var/lib/vami/update/data/job/2/status

If status value is "2", delete the folder (in this case, folder 2 so /opt/vmware/var/lib/vami/update/data/job/2/) and retry upgrade. The job ID may be different.

If there is no stale upgrade flag set, proceed with steps below to manually copy packages.

Download VMware Aria Suite Lifecycle 8.x upgrade ISO from VMware customer connect portal.
Mount the ISO in local system.
Revert the VMware Aria Suite Lifecycle 8.x snapshot to pre-upgrade state.
In VMware Aria Suite Lifecycle 8.x appliance, create a directory 'packages' under the /data directory.
```
mkdir /data/packages
```
Enable the debug mode for the vami log by running the following command
```
sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml
```
Locate the missing packages (step 4 of debugging section) from the already mounted VMware Aria Suite Lifecycle 8.x upgrade ISO
Copy the missing package to '/data/packages' directory inside VMware Aria Suite Lifecycle 8.x VM from the ISO.
Note: Use FTP tool to move the file from the mounted ISO in local system to VMware Aria Suite Lifecycle 8.x VM.
Take a new snapshot of VMware Aria Suite Lifecycle 8.x at this stage (even if the other clean pre-upgrade snapshot exists, taking a snapshot here may save time).
Open one SSH session (let us call this ssh-session-1 for reference in this document) to vRSLCM appliance and keep the following command ready (do not execute because it will fail if executed before the upgrade starts)
```
cp -a /data/packages/. /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/
```
Launch another SSH session (let us call this session-2 for reference in this document) to same vRSLCM appliance and tail the vami log.
```
tail -f /opt/vmware/var/log/vami/vami.log
```
Start the VMware Aria Suite Lifecycle 8.x upgrade from the VMware Aria Suite Lifecycle 8.x UI.
In the ssh-session-1, keep trying on to execute the command (indicated in step 10). Note: Since it is difficult to execute the command exactly at the correct moment, it is better to attempt the command as soon as the upgrade is started and we need to keep on trying it (execute it once in the ssh-session-1 and then in the same session press up-arrow and press enter for a quick retry) until it executes without error or fails with a different package missing error.
If the command is run properly, then VMware Aria Suite Lifecycle 8.x upgrade shall go through and it will complete successfully. In case the upgrade fails again due to same reason, we need to revert back to the snapshot taken in step (8) and retry steps (9) to (12).
Once the VMware Aria Suite Lifecycle 8.x upgrade completes, disable the debug mode for vami log
```
sed -i 's/debug/info/g' /opt/vmware/etc/vami/vami.xml
```

Additional Information

vRSLCM upgrade to 8.x stuck on downloading packages state