/opt/vmware/support partition shows 100% utilization despite no visible large files in vSphere Replication

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• vSphere Replication plugin show "Low disk space" warning message

• The disk usage on the partition on /opt/vmware/support goes to 100% frequently:

admin@vLR[~]$ df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/support_vg-support  3.9G  3.9G     0 100% /opt/vmware/support
/dev/loop0                      2.3G  2.2G  8.9M 100% /opt/vmware/support/logs/srm/SRAs

• Executing the du -sh * command within the /opt/vmware/support directory reveals no files consuming excessive disk space.

root@vr_hostname [ /opt/vmware/support ]# du -sh *
4.0K    hms
4.0K    impex
352M     logs
16K     lost+found

Environment

• VMware Live Recovery 9.0.3
• VMware Live Recovery 9.0.4
• VMware Live Recovery 9.0.5

Cause

• The underlying cause is identified as an issue with the log rotation mechanism. The vSphere Replication Management Server (HMS) Java process retains open file handles on log files that are rotated and deleted by the filesystem. This results in "ghost" files that continue to grow in memory and consume disk space, as the space is unable to be released until the parent process is terminated.
• The condition is validated by executing the lsof command on the appliance, which confirms that the HMS process is actively holding file descriptors for .tar.gz files marked as deleted.

lsof | grep -i deleted

COMMAND       PID     TID TASKCMD              USER   FD      TYPE               DEVICE  SIZE/OFF     NODE  NAME
gzip        7575                               vrms    3r      DIR               254,1         0      40965 /opt/vmware/support/hms/HSB-hms-8c030f47-4a4c-4dc6-9b36-########### (deleted)
gzip        7575                               vrms    4r      REG               254,1 651161600      40966 /opt/vmware/support/hms/HSB-hms-8c030f47-4a4c-4dc6-9b36-###########/bundle.tar.gz__tmp_1782831816181 (deleted)
gzip        9020                               vrms    3r      DIR               254,1         0      65537 /opt/vmware/support/hms/HSB-hms-0a6ac6d2-5b6b-4fa4-be07-########### (deleted)
gzip        9020                               vrms    4r      REG               254,1 654376960      65538 /opt/vmware/support/hms/HSB-hms-0a6ac6d2-5b6b-4fa4-be07-###########/bundle.tar.gz__tmp_1782832235399 (deleted)

Resolution

To resolve the immediate storage constraint, apply the following workaround to stop the HMS service, clear the locked log files, and modify the log rotation configuration to truncate files in place.

1. Establish an SSH connection to the vSphere Replication appliance as the "admin" user.
2. Change user to "root" by running command su -
3. Stop the HMS service by executing the following command: systemctl stop hms
4. Navigate to the support directory and remove the stale log files:
   1. cd /opt/vmware/support/logs/hms
   2. rm hms-stderr.log*
   3. rm hms-stdout.log*
5. Open the log rotation configuration file in a text editor: vi /etc/logrotate.d/hms-service.conf
   • /opt/vmware/logs/hms/hms-stdout.log /opt/vmware/logs/hms/hms-stderr.log {
   size 50M
   rotate 5
   compress
   nocreate
   nodateext
   su root vrms
}
6. Modify the configuration block by replacing the nocreate parameter with copytruncate and updating the file paths to match the environment. Ensure the updated block reflects the following configuration:
   • /var/log/vmware/hms/hms-stdout.log /var/log/vmware/hms/hms-stderr.log {
   size 50M
   rotate 5
   compress
   copytruncate
   nodateext
   su root vrms
}
7. Start the HMS service by running command systemctl start hms
8. Restart the logrotate service by running command systemctl restart logrotate

Additional Information

vSphere Replication 9.0.5 Release Notes - The /opt/vmware/support folder gets full in several days

Protection and Recovery 9.1 Release Notes - The /opt/vmware/support folder gets full in several days