/dev/mapper/support_vg-heapdump partition may appear as full in vSphere replication appliance.

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

The /dev/mapper/support_vg-heapdump partition may appear as full while executing the command "df -h"  in  vSphere replication appliance.

Filesystem                       Size  Used Avail Use% Mounted on
devtmpfs                         3.9G     0  3.9G   0% /dev
tmpfs                            3.9G   28K  3.9G   1% /dev/shm
tmpfs                            3.9G  740K  3.9G   1% /run
tmpfs                            3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda4                         14G  2.7G   11G  21% /
tmpfs                            3.9G  164K  3.9G   1% /tmp
/dev/sda2                        238M   25M  201M  11% /boot
/dev/mapper/support_vg-logs      975M  377M  548M  41% /opt/vmware/logs
/dev/mapper/support_vg-vrmsdb    9.8G   79M  9.2G   1% /var/lib/vrmsdb
/dev/mapper/support_vg-support   2.0G  462M  1.4G  26% /opt/vmware/support
/dev/mapper/support_vg-heapdump  3.9G  3.7G     0 100% /opt/vmware/heapdump
tmpfs                            797M     0  797M   0% /run/user/666

Environment

vSphere Live Recovery 9.x

Cause

• The java heap space appears to be out of memory due to large number of hprof files filling up the space.

• root@replication  /opt/vmware/heapdump/hms ]# ls -ltr

  total 3844944

  -rw------- 1 vrms vrms 2544106258 Oct  6 06:08 java_pid791825.hprof

  -rw------- 1 vrms vrms 1389260800 Oct  6 09:26 java_pid829866.hprof

  -rw------- 1 vrms vrms          0 Oct  6 12:36 java_pid855021.hprof

  -rw------- 1 vrms vrms          0 Oct  6 14:58 java_pid880868.hprof

  -rw------- 1 vrms vrms          0 Oct  6 18:12 java_pid914782.hprof

  -rw------- 1 vrms vrms          0 Oct  6 20:31 java_pid947330.hprof

  -rw------- 1 vrms vrms          0 Oct  6 23:45 java_pid977528.hprof

  -rw------- 1 vrms vrms          0 Oct  7 01:28 java_pid1000811.hprof

  -rw------- 1 vrms vrms          0 Oct  7 04:35 java_pid1024425.hprof

  -rw------- 1 vrms vrms          0 Oct  7 06:16 java_pid1052017.hprof

     /opt/vmware/logs/hms/hms-stdout.log

     Caused by: java.lang.OutOfMemoryError: Java heap space

     java.lang.OutOfMemoryError: Java heap space

     Dumping heap to /opt/vmware/heapdump/hms/java_pid977528.hprof ...

Resolution

1)  Run lsblk command to confirm that the vSphere Replication DB is on the second disk, and on a logical volume:

2)  Confirm the LVM physical volume size: pvs

3)  Increase the disk on the vSphere Replication VM from 17 GB to, say, 20 GB. (You will not be able to increase if snapshots are present)

4)  Resize the physical volume: pvresize /dev/sdb

5)  Reboot the vSphere Replication appliance

6) Re-run pvs command, to show added free space: lvdisplay

    The df -h command may report  /dev/mapper/support_vg-heapdump is full, use the command lvextend -L6G /dev/mapper/support_vg-heapdump

    Where -L3G means that the final size of the LVM would be around 6GB, so increase it as per your need. -L indicates the final size of the volume.

7) Resize the file system:

    resize2fs /dev/mapper/support_vg-heapdump

8) Confirm available, used and free space by re-running df -h command
 
9) Reboot the VR appliance and from the VAMI complete "Save and restart service".