"/opt/vmware/support" directory frequently becomes full within the VMware Live Recovery Appliance

Products

• VMware Live Recovery

Issue/Introduction

• Space constraints are noticed inside the "/opt/vmware/support" folder within the VMware Live Recovery Appliance
• The directory and the partition often gets full
• /opt/vmware/support/logs/hms directory has a list of big files

Environment

VMware Live recovery Appliance 9.0.3 and above

 

Cause

• The hms-stderr.log file is deleted by logrotate but still referenced by the hms process.
• The HMS process holds a massive log file that was technically "deleted" from the file system.
• Because the process is still running, the disk space isn't actually released, causing the "ghost" file to grow indefinitely.

Resolution

This issue us fixed in Protection and Recovery 9.1.1

Workaround:

1. Login into the VLR appliance using "admin" credentials.
2. Change user to "root".
3. Stop the HMS service: systemctl stop hms
4. Navigate to /opt/vmware/support/logs/hms
5. Verify the (deleted) files by running: lsof | grep deleted
6. Remove all hms-stderr.log* and hms-stdout.log* files by running:
   rm hms-stderr.log*
rm hms-stdout.log*
7. Navigate to /etc/logrotate.d/

8. Edit file hms-service.conf to remove "nocreate" and add "copytruncate" as per below:
   vi hms-service.conf

   /var/log/vmware/hms/hms-stdout.log
   /var/log/vmware/hms/hms-stderr.log {
     size 50M
     rotate 5
     compress
     copytruncate
     nodateext
     su root vrms
   }

9. Verify the change
   cat /etc/logrotate.d/hms-service.conf
10. Start the HMS service systemctl start hms
11. Restart the logrotate service : systemctl restart logrotate