vSphere Replication 6.x and 8.x service fails to start, the embedded database volume is 100% full - Expanding logical volume

Products

• VMware Live Recovery
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

On vSphere Replication 6.5 and 8.1, the hms service fails to start because the logical volume containing the embedded database is 100% full

"df" command confirms that /dev/mapper/support_vg-vrmsdb is 100% full

Environment

VMware vSphere Replication 8.x
VMware vSphere Replication 6.5.x

Resolution

1. Run lsblk command to confirm that the vSphere Replication DB is on the second disk, and on a logical volume:

(Note: the 2nd disk on a VR 6.5 or 8.1 appliance is 13 GB on deployment. The above shows a second disk already extended to 20 GB.)


2. Confirm the LVM physical volume size:

# pvs
PV            VG             Fmt     Attr   PSize        PFree
/dev/sdb   support_vg lvm2   a--    <13.00g     0

3. Increase the disk on the vSphere Replication VM from 13 GB to, say, 20 GB.

4. Resize the physical volume:

# pvresize /dev/sdb

5. Take a snapshot of the vSphere Replication VM

6. Reboot the vSphere Replication appliance

7. Re-run pvs command, to show added free space:

 

 

8. Confirm the logical volume path and extend it:

# lvdisplay
...
 LV Path                /dev/support_vg/vrmsdb
...
# lvextend -l +100%FREE /dev/support_vg/vrmsdb

e.g.

 

In some cases, both /dev/mapper/support_vg-vrmsdb and /dev/mapper/support_vg-logs might be 100% full. then the added free space will be shared between the logical volumes, e.g.:

# lvextend -l +80%FREE /dev/support_vg/vrmsdb
# lvextend -l +20%FREE /dev/support_vg/logs

9. Resize the file system:

# resize2fs /dev/support_vg/vrmsdb

10. Confirm available, used and free space by re-running df command

11. Reboot the VR appliance and from the VAMI complete "Save and restart service". 

12. When the appliance is confirmed to be running stably, consolidate the snapshot.

Additional Information

Internal KB "Root Partition 100% vSphere Replication 5.8.0 (2110026)" provides a similar solution for vSphere Replication 5.8 (applies to versions up to 6.1.x)

Impact/Risks:
Low risk.