vRSLCM upgrade fails or UI is inaccessible due to the root partition being full

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Aria Suite Lifecycle Manager upgrade fails.
  • OR
• UI is inaccessible.

 

Environment

VMware Aria Suite Lifecycle Manager 8.x

Cause

This issue occurs when the root partition on the vRSLCM appliance is full and does not have enough space to load the vRSLCM RPM packages.

• df -h command displays similar result on /dev/sda4 partition:

Filesystem           Size Used Avail Use% Mounted on
devtmpfs             2.9G   0 2.9G  0% /dev
tmpfs               3.0G  20K 3.0G  1% /dev/shm
tmpfs               3.0G 1.1M 3.0G  1% /run
tmpfs               3.0G   0 3.0G  0% /sys/fs/cgroup
/dev/sda4             8.7G 7.0G 2.0G 85% /
tmpfs               3.0G  68K 3.0G  1% /tmp
/dev/sda2             119M  28M  85M 25% /boot
/dev/mapper/stroage_vg-storage 9.8G 156M 9.2G  2% /storage
/dev/mapper/data_vg-data     49G  21G 5.6G 39% /data
tmpfs               595M   0 595M  0% /run/user/0

Resolution

This issue can be resolved by following the below steps on the appliance.

Note: If the upgrade has failed, revert it to the snapshot and then perform the below cleanup steps.

1. Take a non-memory snapshot.
2. SSH to LCM appliance.
3. Run the below command: 

find / -xdev -type f -size +100M -exec du -sh {} ';' | sort -rh | head -n50

Example output:

1.8G  /var/lib/loginsight-agent/storage/liagent.db
303M  /var/log/nginx/access.log
244M  /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/blackstone-8.12.0.7-21628952.noarch.rpm
233M  /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vlcm-files-8.12.0.7-21628952.noarch.rpm
221M  /opt/vmware/vlcm/postgresbackups/data-backup-2023-03-29-05:22:56.backup
186M  /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vlcm-8.12.0.7-21628952.noarch.rpm
173M  /var/lib/vrlcm/vmlcm-service-old-SNAPSHOT.jar_bkp
173M  /var/lib/vrlcm/vmlcm-service-8.10.0-SNAPSHOT.jar
157M  /var/log/sa
148M  /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/VMware-commonjars-7.0.3.00000-9019451.x86_64.rpm
117M  /opt/vmware/vlcm/blackstone_bkp/spring-common/blackstone-external-8.10.0.jar
117M  /opt/vmware/vlcm/blackstone/spring-common/blackstone-external-8.10.0.jar

Note: You can see the liagent.db which is taking up ~ 1.8 GB.

4. Change directory to:

cd /var/lib/loginsight-agent/storage

5. Check the status of the service. It should be active/running:

systemctl status liagentd.service  

6. Stop liagentd service:

systemctl stop liagentd.service

7. Remove the liagent.db file.

rm liagent.db

8. Start the liagent service:

systemctl start liagentd.service

Verify the space on / by running df -h It should be more than 3 GB.

9. Reboot the appliance:

reboot -f

Once VRSLCM is up and running, re-trigger the VRSLCM upgrade.

Additional Information

If the above actions do not reclaim enough space. Do follow below KB article to extend the File System.

Reference KB: Increase root '/' partition on VMware Aria Suite Lifecycle 8.x