'Low disk space' or 'Insufficient disk space' warning for vSphere Replication on vCenter

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• vSphere Replication will occasionally report low disk space or insufficient disk space when the log partition fills up.

• This disk exhaustion may lead to widespread RPO violations as the appliance lacks the necessary buffer space to process replication metadata and write operation logs.

Environment

VMware Live Recovery 9.x

vSphere Replication 8.x

Cause

Potential causes include:

• vSphere Replication Server log bundles not being cleared after generation.

• Very high frequency events filling up logs

• Services such as the Apache Tomcat Java Servlet service failing to clean up files

• /storage/log partition set too small.

• The vSphere Replication log partition is 100% full.

  

Resolution

Deleting old log files in the appliance will free up space and remove the notification from the plugin in vCenter.  

NOTE: Take a memory snapshot of the appliance before performing the steps below. 

WARNING: Exercise caution when running the rm commands because once you delete a file it cannot be recovered. The rm command removes the entries for a particular file, group of files, or certain select files from a list within a directory. When you use the rm command, user confirmation, read permission, and write permission is not necessary before a file is deleted.

Steps to clean up the old files:

• Check the file usage in the appliance by running the df -h command.

Example as below.

• It's safe and sufficient to delete the files mentioned below to free-up disk space in the /opt/vmware/logs and /opt/vmware/support partition. 

Example as below:

root@vsphere-replication [ /var/log/journal ]# ls

6f57f5c608a24f2ba0d45b9ffbf97291  808021b8da5445d8927f06a9a0b3059b  log  

remoteroot@vsphere-replication [ /var/log/journal ]# rm -rf *

root@vsphere-replication [ /opt/vmware/logs/hms ]# ls

rm -rf hms.*.log.gz
rm -rf hms-audit.*.log.gz
rm -rf hms-change.*.log.gz
rm -rf hms-eventlog.*.log.gz
rm -rf hms-hks.*.log.gz

root@vsphere-replication [ /opt/vmware/support/hms ]# ls

rm -rf hsb-*.gz

root@vsphere-replication [ /var/log/vmware ]# ls

rm -rf hbrsrv-*-int.stats.gz
rm -rf hbrsrv-*.stats
rm -rf hbrsrv-*.log.gz

root@vsphere-replication [ /var/log/vmware/dr ]# ls

rm -rf drconfig-*.log.gz

root@vsphere-replication [ /var/log/vmware/envoy ]# ls

rm -rf envoy-*.log.gz

• If a log file is to be emptied and not deleted, the following command can be run: echo > filename

• Reboot the appliance and check the disk space now to see how much space has been released.

Example as below.

• Ensure that all essential services are running and delete the snapshot after a while.

Important : If the low disk space alert persists even after deleting the log files, please contact Broadcom Support for further investigation.