Aria Suite Lifecycle UI does not load and Postgres service is down because /storage is full
search cancel

Aria Suite Lifecycle UI does not load and Postgres service is down because /storage is full

book

Article ID: 345999

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Aria Suite Lifecycle UI does not load and UI is stuck at "Waiting for services to start".
  • When attempting to log into the Aria Suite Lifecycle UI, the log in fails with Error "Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection "
  • vrlcm-cli --health-status shows, psotgres service was unhealthy and not running on vASL
  • The services on the appliance will not start and the UI will be down. 
  • journalctl -x shows postgres service failures due to lack of available device space.
  • The postgres service on the Aria Suite Lifecycle appliance is down.
  • /storage partition on the Aria Suite Lifecycle appliance is full at 100% utilization.

Environment

VMware Aria Suite Lifecycle 8.x

Cause

This happens because the /storage on the Aria Suite Lifecycle appliance is full and does not have enough space to load the Aria Suite Lifecycle services.

Resolution

  •  To restore the functionality of the Aria Suite Lifecycle, increase the free space on the '/storage' partition of Aria Suite Lifecycle appliance.
    1. Increasing disk size of '/storage' partition: 
      • This would be the preferred method if '/storage' is at 100% utilization, as the postgres service may be unhealthy to access db and clear table entries.
      • Note: The disk cannot be increased when a snapshot exists on the virtual machine. It is recommended to have daily backups of the system using any supported 3rd party utility that can manage vSphere snapshots as a backup provider before proceeding with these changes.   
        1. Login using root credentials to the Aria Suite Lifecycle Appliance using SSH. 
        2. Check the disk space utilization consuming high space: df -h. Validate that '/storage' partition is at high utilization. 
        3. Run either of the below commands to identify the disk on which "/storage" is configured:
          • lsblk

            NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
            sda                                 8:0    0 10.6G  0 disk
            ├─sda1                              8:1    0    4M  0 part
            ├─sda2                              8:2    0   10M  0 part /boot/efi
            ├─sda3                              8:3    0  512M  0 part /boot
            └─sda4                              8:4    0   10G  0 part
             └─system-system_0               254:5    0   10G  0 lvm  /
            sdb                                 8:16   0 50.1G  0 disk
            └─data-data_0                     254:4    0   50G  0 lvm  /data
            sdc                                 8:32   0 10.1G  0 disk
            └─storage-storage_0               254:3    0   10G  0 lvm  /storage
            sdd                                 8:48   0  8.1G  0 disk
            └─swap-swap_0                     254:2    0    8G  0 lvm  [SWAP]
            sde                                 8:64   0 10.1G  0 disk
            └─vg_alt_root-lv_alt_root         254:1    0   10G  0 lvm  /storage/alt_root
            sdf                                 8:80   0  8.1G  0 disk
            └─vg_lvm_snapshot-lv_lvm_snapshot 254:0    0    8G  0 lvm  /storage/lvm_snapshot
            sr0                                11:0    1 1024M  0 rom
          • lvdisplay
             --- Physical volume ---
             PV Name               /dev/sdc
             VG Name               storage
             PV Size               10.06 GiB / not usable 4.00 MiB
             Allocatable           yes
             PE Size               4.00 MiB
             Total PE              2575
             Free PE               15
             Allocated PE          2560
             LV UUID               [UUID]
          • pvdisplay
            --- Logical volume ---
             LV Path                /dev/storage/storage_0
             LV Name                storage_0
             VG Name                storage
             LV UUID                [UUID]
             LV Write Access        read/write
             LV Creation host, time photon, 2024-05-10 09:13:55 +0000
             LV Status              available
             # open                 1
             LV Size                10.00 GiB
             Current LE             2560
             Segments               1
             Allocation             inherit
             Read ahead sectors     auto
             - currently set to     256
             Block device           254:3

            The '/storage' partition is usually mounted on the /dev/sdc.
            Note: Incase there are 2 disks with same storage size check by running lsscsi -v , this will show controller ID.

        4. Log in to vCenter and locate the Aria Suite Lifecycle virtual machine.
        5. Shutdown Aria Suite Lifecycle using a graceful Guest Operating system shut down.
        6. Right-click on the Aria Suite Lifecycle appliance and select Edit Settings.
        7. Navigate to the Hard Disk section and increase the size (By 10Gb More) of the disk that corresponds to the full partition.
        8. Added more disk space to this drive from vCenter.


          From the above examples, it can be identified that the /storage is /dev/sdc - Hence, to increase the '/storage' partition space, the /dev/sdc needs to be extended, in this case, Hard disk 3. 

        9. Save the changes and power on the Aria Suite Lifecycle appliance.
          When rebooted, the autogrow script detects the change in the logical volume and increases the size automatically.
          • If the partitions are not updated after the reboot, manually run /etc/bootstrap/everyboot.d/20-autogrow-disk.
        10. Log in to the Aria Suite Lifecycle Appliance using root credentials and run the command to validate that the root partition has been increased: df -h
        11. Run the command to validate that all the services are running as expected:
          • root@vrlcm [ - ] #vrlcm-cli -- health-status
            bootstrap ------------------------ SUCCESS
            postgres ------------------------- Running
            vrlcm-server -------------------- Running
            blackstone ---------------------- Running
        12. Validate that the Aria Suite Lifecycle UI should be accessible now.

    2. Cleaning up tables in the Aria Suite Lifecycle DB:
      • This method would only be possible if the postgres service is running and the '/storage' partition is not at 100% utilization. 
        1. Ensure valid snapshot and backups are in place.
        2. Log in to the Aria Suite Lifecycle Appliance using root credentials.
        3. Connect to the Aria Suite Lifecycle DB:
          su - postgres
          cd /opt/vmware/vpostgres/11/bin
          ./psql -d vrlcm

        4. Run VACUUM FULL VERBOSE ANALYZE vm_rs_request; to clean up DB.
           
        5. run \q to quit the database 
        6. exit the psql bash.
        7. Validate that the UI is accessible.
  • if issues persist, follow instructions as outlined in Increase root '/' partition on VMware Aria Suite Lifecycle 8.x

 

Note: All screenshots are from lab environment 

  1.  

Additional Information

NOTE: If the new space is still not showing after running the script manually in Step 9, you may need to ensure that it is not in use before running the /etc/bootstrap/everyboot.d/20-autogrow-disk script manually, as per https://knowledge.broadcom.com/external/article/398442

Powered by Wolken Software