UI is inaccessible for extended period during patching of Aria Suite Lifecycle.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to upgrade Aria Suite Lifecycle Manager, the upgrade process fails or the User Interface (UI) becomes inaccessible.
  
  
• Aria Suite Lifecycle Manager UI does not load and UI is stuck at "Waiting for services to start"

• • df -h command displays similar result on /dev/sda4 partition:

  Filesystem           Size Used Avail Use% Mounted on
  devtmpfs             2.9G   0 2.9G  0% /dev
  tmpfs               3.0G  20K 3.0G  1% /dev/shm
  tmpfs               3.0G 1.1M 3.0G  1% /run
  tmpfs               3.0G   0 3.0G  0% /sys/fs/cgroup
  /dev/sda4             8.7G 7.0G 2.0G 85% /
  tmpfs               3.0G  68K 3.0G  1% /tmp
  /dev/sda2             119M  28M  85M 25% /boot
  /dev/mapper/stroage_vg-storage 9.8G 156M 9.2G  2% /storage
  /dev/mapper/data_vg-data     49G  21G 5.6G 39% /data
  tmpfs               595M   0 595M  0% /run/user/0

• The command vrlcm-cli --health-status displays the health status of the appliance as:
  
  bootstrap ------------- In progress
postgres ------------- Not Running
vrlcm-server -------------Not Running
blackstone ------------- Not Running
  
  
• While checking the system logs, the /var/log/vrlcm/patchcli.log may incorrectly indicate that the patch installation process has completed successfully:
  
  __main__ - INFO - patch installation process ended.
__main__ - INFO - patch installation completed.


• However, a review of the /var/log/vrlcm/os-package-update.log reveals that the underlying OS upgrade from Photon OS 3.0 to Photon OS 5.0 has failed, specifically displaying a tdnf error code: 245:
  
  Error in upgrading to Photon OS 5.0 from 3.0 ((tdnf error code: 245).
Original list of RPMs and RPM DB are stored in /tmp/photon-upgrade-xC8,  please provide contents of that folder along with system journal   logs for analysis; these logs can be captured using command-
# /usr/bin/journalctl -xa > /tmp/photon-upgrade-xC8/journal.log
Cannot continue. Aborting.
  
  
• Alternatively the /var/log/vrlcm/os-package-update.log file reports the lack of filesystem space with the following messages:
  • Found 5 problems
installing package openjdk11-jre-11.0.30-2.ph5.x86_64 needs 22MB more space on the / filesystem
installing package openjdk11-11.0.30-2.ph5.x86_64 needs 28MB more space on the / filesystem
installing package apache-ant-1.10.12-8.ph5.noarch needs 32MB more space on the / filesystem
installing package apache-tomcat10-10.1.53-2.ph5.noarch needs 45MB more space on the / filesystem
installing package python3-psycopg2-2.9.3-7.1.ph5.x86_64 needs 46MB more space on the / filesystem
Error(1525) : rpm transaction failed
  • Error installing replacement and earlier removed packages.
Original list of RPMs and RPM DB are stored in /tmp/photon-upgrade-Dvu,  please provide contents of that folder along with system journal logs for analysis; these logs can be captured using command-
 # /usr/bin/journalctl -xa > /tmp/photon-upgrade-Dvu/journal.log
 Cannot continue. Aborting.

Environment

VMware Aria Suite Lifecycle Manager 8.x

Cause

The root partition on the Aria Suite Lifecycle appliance requires 3GB to load the RPM packages during the patching process. This issue is specific to instances where LCM is being upgraded from Patch 3 to later patches, which upgrade Photon OS from version 3 to 5.

 

Resolution

Please raise a Broadcom Support Request noting this Article ID (318504) in the problem description. For more information, see Creating and managing Broadcom support cases

Additional Information

Extend root and alt_root volumes on VMware Aria Suite Lifecycle appliance for versions 8.12.x and later

If the partition type is part and not LVM for /dev/sda4 use this KB to extend the drive: 

• How to increase /dev/sda4 partition space in Aria Suite Lifecycle