vSphere Replication 8.x service fails to start, the embedded database volume or the log folder is 100% full - Expanding logical volume

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

On vSphere Replication 8.x and 9.x, the hms service fails to start because the logical volume containing the embedded database is 100% full

"df -h" command confirms that /dev/mapper/support_vg-vrmsdb or the dev/mapper/support_vg-logs is 100% full

Environment

VMware vSphere Replication 8.x
VMware vSphere Replication 9.x

Resolution

1. Run lsblk command to confirm that the vSphere Replication DB is on the second disk, and on a logical volume:

(Note: the 2nd disk on a VR 8.x or 9.x appliance is 13 GB on deployment. The above shows a second disk already extended to 20 GB.)


2. Confirm the LVM physical volume size:

# pvs
PV            VG             Fmt     Attr   PSize        PFree
/dev/sdb   support_vg lvm2   a--    <13.00g     0

3. Increase the disk on the vSphere Replication VM from 13 GB to, say, 20 GB.

4. Resize the physical volume:

# pvresize /dev/sdb

5. Reboot the vSphere Replication appliance

6.  Re-run pvs command, to show added free space:

 

 

7. Confirm the logical volume path and extend it:

# lvdisplay
...
 LV Path                /dev/support_vg/vrmsdb
...
# lvextend -l +100%FREE /dev/support_vg/vrmsdb

e.g.

 

In some cases, both /dev/mapper/support_vg-vrmsdb and /dev/mapper/support_vg-logs might be 100% full. then the added free space will be shared between the logical volumes, e.g.:

# lvextend -l +80%FREE /dev/support_vg/vrmsdb
# lvextend -l +20%FREE /dev/support_vg/logs

8. Resize the file system:

# resize2fs /dev/support_vg/vrmsdb

9. Confirm available, used and free space by re-running df -h command

 

10. Reboot the VR appliance and from the VAMI complete "Save and restart service". 

Additional Information

The /dev/sda2 is a boot partition. There is no need to extend its size usually.

The /opt/vmware/support/logs partition can also be cleaned up to create more free space using 'Low disk space' or 'Insufficient disk space' warning for vSphere Replication on vCenter

But if we notice support_vg-support (/opt/vmware/logs) running out of space repeatedly.

Customer may need to extend the size for below LVM volumes.

/dev/sda2                        238M   36M  190M  16% /boot
/dev/mapper/support_vg-logs      975M  374M  551M  41% /opt/vmware/logs
/dev/mapper/support_vg-heapdump  3.9G   96K  3.7G   1% /opt/vmware/heapdump
/dev/mapper/support_vg-support   2.0G  232M  1.6G  13% /opt/vmware/support
/dev/mapper/support_vg-vrmsdb     64G  557M   61G   1% /var/lib/vrmsdb

We can use the same steps to extend the other LVM volumes support_vg-support (/opt/vmware/logs)  & support_vg-logs (/opt/vmware/support).