Troubleshooting VMware Aria Suite Lifecycle 8.x upgrade failures due to missing packages
search cancel

Troubleshooting VMware Aria Suite Lifecycle 8.x upgrade failures due to missing packages

book

Article ID: 326076

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
The VMware Aria Suite Lifecycle 8.x upgrade failure can be caused due to many other symptoms as well but for this KB the relevant ones are as follows
  • You trigger VMware Aria Suite Lifecycle 8.x upgrade and after running through a few of the initial steps, the upgrade progress in UI seems to be frozen. If left alone, the UI remains at this stage for hours.
  • Similar errors to the below are found in the /opt/vmware/var/log/vami/vami.log
12/09/2022 09:39:20 [INFO] Using update pre-install script
12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing


Environment

VMware Aria Suite Lifecycle 8.x
VMware vRealize Suite Lifecycle Manager 8.x

Cause

The VMware Aria Suite Lifecycle 8.x upgrade process fails to capture the required binary files from the upgrade source. Both online and CD-ROM upgrade source methods are impacted.

 

 

Resolution

See the Workaround section below for additional information.

Workaround:

Prerequisites

  • Please take simultaneous non-memory snapshots of the virtual appliance(s).
  • You have access to root user and password.
  • You have SSH or console access to the virtual appliance.
  • Download the upgrade ISO from VMware Customer Connect.

Troubleshooting Procedure

  1. Revert the appliance to its pre-upgrade snapshot.
  2. Enable the debug mode for the vami log by running the following command 
    sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml
  3. Tail the vami.log using the command below, then re-run the upgrade (like it was done initially when the failure first occurred) 
    tail -f /opt/vmware/var/log/vami/vami.log
  4. In debug mode, the missing packages will be listed. Record this information for use in later steps. See below for an example of a missing package

    14/09/2022 10:42:34 [DEBUG] Stat file, file missing. filename=/opt/vmware/var/lib/vami/update/data/package-pool/package-pool/audit-2.8.5-17.ph3.x86_64.rpm

Workaround

  1. As a first step check for the existence of a stale upgrade status file.

    vami.log will show below

    27/10/2022 11:17:15 [DEBUG] Lock file 1
    27/10/2022 11:17:15 [DEBUG] status file /opt/vmware/var/lib/vami/update/data/job/2/status
    27/10/2022 11:17:15 [DEBUG] cli failed
    27/10/2022 11:17:15 [DEBUG] cache path is/opt/vmware/var/lib/vami/update/data
    27/10/2022 11:17:15 [DEBUG] No next version exists
    27/10/2022 11:17:15 [DEBUG] No next version exists
    27/10/2022 11:17:15 [DEBUG] Setting download cache location: /opt/vmware/var/lib/vami/update/data
    27/10/2022 11:17:15 [DEBUG] Disabling old download cache location:
    27/10/2022 11:17:15 [DEBUG] Done setting download cache location
    27/10/2022 11:17:15 [DEBUG] skipClearDownloadCache =
    27/10/2022 11:17:15 [DEBUG] Clear Download Cache. path=/opt/vmware/var/lib/vami/update/data

    and upgrade again fails with below error:

    12/09/2022 09:39:20 [ERROR] Unable to install updates because package files are missing

    Take the erroneous job ID from above snippet and go to that folder and check contents of status file.

    less /opt/vmware/var/lib/vami/update/data/job/2/status
    

    If status value is "2", delete the folder (in this case, folder 2 so /opt/vmware/var/lib/vami/update/data/job/2/) and retry upgrade. The job ID may be different.

    If there is no stale upgrade flag set, proceed with steps below to manually copy packages.

  2. Download VMware Aria Suite Lifecycle 8.x upgrade ISO from VMware customer connect portal.
  3. Mount the ISO in local system.
  4. Revert the VMware Aria Suite Lifecycle 8.x snapshot to pre-upgrade state.
  5. In VMware Aria Suite Lifecycle 8.x appliance, create a directory 'packages' under the /data directory. 
    mkdir /data/packages
  6. Enable the debug mode for the vami log by running the following command 
    sed -i 's/info/debug/g' /opt/vmware/etc/vami/vami.xml
  7. Locate the missing packages (step 4 of debugging section) from the already mounted VMware Aria Suite Lifecycle 8.x upgrade ISO        
  8. Copy the missing package to '/data/packages' directory inside VMware Aria Suite Lifecycle 8.x VM from the ISO.
     Note: Use FTP tool to move the file from the mounted ISO in local system to VMware Aria Suite Lifecycle 8.x VM.
  9.  Take a new snapshot of VMware Aria Suite Lifecycle 8.x at this stage (even if the other clean pre-upgrade snapshot exists, taking a snapshot here may save time). 
  10. Open one SSH session (let us call this ssh-session-1 for reference in this document) to vRSLCM appliance and keep the following command ready (do not execute because it will fail if executed before the upgrade starts)  
    cp -a /data/packages/. /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/
  11. Launch another SSH session (let us call this session-2 for reference in this document) to same vRSLCM appliance and tail the vami log. 
    tail -f /opt/vmware/var/log/vami/vami.log
  12. Start the VMware Aria Suite Lifecycle 8.x upgrade from the VMware Aria Suite Lifecycle 8.x UI.
  13. In the ssh-session-1, keep trying on to execute the command (indicated in step 10). Note: Since it is difficult to execute the command exactly at the correct moment, it is better to attempt the command as soon as the upgrade is started and we need to keep on trying it (execute it once in the ssh-session-1 and then in the same session press up-arrow and press enter for a quick retry) until it executes without error or fails with a different package missing error. 
  14. If the command is run properly, then VMware Aria Suite Lifecycle 8.x upgrade shall go through and it will complete successfully. In case the upgrade fails again due to same reason, we need to revert back to the snapshot taken in step (8) and retry steps (9) to (12).
  15. Once the VMware Aria Suite Lifecycle 8.x upgrade completes, disable the debug mode for vami log 
    sed -i 's/debug/info/g' /opt/vmware/etc/vami/vami.xml

 


Additional Information

vRSLCM upgrade to 8.x stuck on downloading packages state