Replication Failures Due to /dev/sda4 Partition Full in vSphere Replication Appliance

Products

• VMware Live Recovery

Issue/Introduction

Symptoms 

• Replication failures observed in the vSphere Replication environment

Validation Steps :  

• SSH into the vSphere Replication appliance.

• Run df -h and verify the usage of the /dev/sda4 partition:

       

• If the partition is 100% full, proceed to check the log files.

• Check the size of /var/log/localmessages:

  du -h /var/log/* | sort -h

 

 

Environment

VMware Live Recoery : 9.x

Cause

The issue occurs due to uncontrolled growth of the /var/log/localmessages log file. Since log rotation was not configured for this custom log file, it continued to consume disk space on the /dev/sda4 partition. Once the partition became full, SQLite operations failed, triggering replication errors

Cause validation

• Repeated SQLite database errors appear in the hbrsrv.log.  :  /var/log/hbrsrv.log in Vsphere replication appliance
  
  2025-04-27T18:55:53.295066+00:00 h01vr01 hbrsrv[641873]: 2025-04-27T18:55:53.294Z error hbrsrv[646153] [Originator@6876 sub=Main groupID=GID-####-###-4c53-###-###opID=hsl-4486b5a7] HbrError stack:
  2025-04-27T18:55:53.295111+00:00 h01vr01 hbrsrv[641873]: 2025-04-27T18:55:53.295Z error hbrsrv[646153] [Originator@6876 sub=Main groupID=GID-###-f731-###-bf44-####opID=hsl-4486b5a7]    [0] SQLite error 13: database or disk is full
  2025-04-27T18:55:53.295148+00:00 h01vr01 hbrsrv[641873]: 2025-04-27T18:55:53.295Z error hbrsrv[646153] [Originator@6876 sub=Main groupID=GID-###-f731-###-bf44-####opID=hsl-4486b5a7]    [1] Returned error message: database or disk is full
  2025-04-27T18:55:53.295186+00:00 h01vr01 hbrsrv[641873]: 2025-04-27T18:55:53.295Z error hbrsrv[646153] [Originator@6876 sub=Main groupID=GID-###-f731-###-bf44-###opID=hsl-4486b5a7]    [2] Add disk instance properties in database (diskID=RDID-####-c75d-###-####-0ac7eb6ab601)

Resolution

• Implement Log Rotation for /var/log/localmessages
• Create a new logrotate configuration file
     sudo vi /etc/logrotate.d/localmessages
• Add the following content:
  /var/log/localmessages { daily
                                                     rotate 7
                                                    compress
                                                   missingok 
                                               notifempty 
                                              create 0640 root root
                       }
• Restart the VRMS appliance to clear any held resources and refresh logging services