Aria Suite Lifecycle UI does not load and Postgres service is down because /storage is full

Products

VMware Aria Suite

Issue/Introduction

Aria Suite Lifecycle UI does not load and UI is stuck at "Waiting for services to start".
When attempting to log into the Aria Suite Lifecycle UI, the log in fails with Error "Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection "
vrlcm-cli --health-status shows, psotgres service was unhealthy and not running on vASL
The services on the appliance will not start and the UI will be down.
journalctl -x shows postgres service failures due to lack of available device space.
The postgres service on the Aria Suite Lifecycle appliance is down.
/storage partition on the Aria Suite Lifecycle appliance is full at 100% utilization.

Environment

VMware Aria Suite Lifecycle 8.x

Cause

This happens because the /storage on the Aria Suite Lifecycle appliance is full and does not have enough space to load the Aria Suite Lifecycle services.

Resolution

To restore the functionality of the Aria Suite Lifecycle, increase the free space on the '/storage' partition of Aria Suite Lifecycle appliance.
1. Increasing disk size of '/storage' partition:
  - This would be the preferred method if '/storage' is at 100% utilization, as the postgres service may be unhealthy to access db and clear table entries.
  - Note: The disk cannot be increased when a snapshot exists on the virtual machine. It is recommended to have daily backups of the system using any supported 3rd party utility that can manage vSphere snapshots as a backup provider before proceeding with these changes.
    1. Login using root credentials to the Aria Suite Lifecycle Appliance using SSH.
    2. Check the disk space utilization consuming high space: df -h. Validate that '/storage' partition is at high utilization.
    3. Run either of the below commands to identify the disk on which "/storage" is configured:
      - lsblk
        
        NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
        sda 8:0 0 10.6G 0 disk
        ├─sda1 8:1 0 4M 0 part
        ├─sda2 8:2 0 10M 0 part /boot/efi
        ├─sda3 8:3 0 512M 0 part /boot
        └─sda4 8:4 0 10G 0 part
        └─system-system_0 254:5 0 10G 0 lvm /
        sdb 8:16 0 50.1G 0 disk
        └─data-data_0 254:4 0 50G 0 lvm /data
        sdc 8:32 0 10.1G 0 disk
        └─storage-storage_0 254:3 0 10G 0 lvm /storage
        sdd 8:48 0 8.1G 0 disk
        └─swap-swap_0 254:2 0 8G 0 lvm [SWAP]
        sde 8:64 0 10.1G 0 disk
        └─vg_alt_root-lv_alt_root 254:1 0 10G 0 lvm /storage/alt_root
        sdf 8:80 0 8.1G 0 disk
        └─vg_lvm_snapshot-lv_lvm_snapshot 254:0 0 8G 0 lvm /storage/lvm_snapshot
        sr0 11:0 1 1024M 0 rom
      - lvdisplay
        --- Physical volume ---
        PV Name /dev/sdc
        VG Name storage
        PV Size 10.06 GiB / not usable 4.00 MiB
        Allocatable yes
        PE Size 4.00 MiB
        Total PE 2575
        Free PE 15
        Allocated PE 2560
        LV UUID [UUID]
      - pvdisplay --- Logical volume --- LV Path /dev/storage/storage_0 LV Name storage_0 VG Name storage LV UUID [UUID] LV Write Access read/write LV Creation host, time photon, 2024-05-10 09:13:55 +0000 LV Status available # open 1 LV Size 10.00 GiB Current LE 2560 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 254:3The '/storage' partition is usually mounted on the /dev/sdc.
        Note: Incase there are 2 disks with same storage size check by running lsscsi -v , this will show controller ID.
    4. Log in to vCenter and locate the Aria Suite Lifecycle virtual machine.
    5. Shutdown Aria Suite Lifecycle using a graceful Guest Operating system shut down.
    6. Right-click on the Aria Suite Lifecycle appliance and select Edit Settings.
    7. Navigate to the Hard Disk section and increase the size (By 10Gb More) of the disk that corresponds to the full partition.
    8. Added more disk space to this drive from vCenter.
      
      From the above examples, it can be identified that the /storage is /dev/sdc - Hence, to increase the '/storage' partition space, the /dev/sdc needs to be extended, in this case, Hard disk 3.
    9. Save the changes and power on the Aria Suite Lifecycle appliance.
      When rebooted, the autogrow script detects the change in the logical volume and increases the size automatically.
      - If the partitions are not updated after the reboot, manually run /etc/bootstrap/everyboot.d/20-autogrow-disk.
    10. Log in to the Aria Suite Lifecycle Appliance using root credentials and run the command to validate that the root partition has been increased: df -h
    11. Run the command to validate that all the services are running as expected:
      - root@vrlcm [ - ] #vrlcm-cli -- health-status
        bootstrap ------------------------ SUCCESS
        postgres ------------------------- Running
        vrlcm-server -------------------- Running
        blackstone ---------------------- Running
    12. Validate that the Aria Suite Lifecycle UI should be accessible now.
2. Cleaning up tables in the Aria Suite Lifecycle DB:
  - This method would only be possible if the postgres service is running and the '/storage' partition is not at 100% utilization.
    1. Ensure valid snapshot and backups are in place.
    2. Log in to the Aria Suite Lifecycle Appliance using root credentials.
    3. Connect to the Aria Suite Lifecycle DB:
      su - postgres
      cd /opt/vmware/vpostgres/11/bin
      ./psql -d vrlcm
    4. Run VACUUM FULL VERBOSE ANALYZE vm_rs_request; to clean up DB.
    5. run \q to quit the database
    6. exit the psql bash.
    7. Validate that the UI is accessible.
if issues persist, follow instructions as outlined in Increase root '/' partition on VMware Aria Suite Lifecycle 8.x

Note: All screenshots are from lab environment

Additional Information

NOTE: If the new space is still not showing after running the script manually in Step 9, you may need to ensure that it is not in use before running the /etc/bootstrap/everyboot.d/20-autogrow-disk script manually, as per https://knowledge.broadcom.com/external/article/398442