SRM - Storage Replication Adapter Failed

Products

• VMware Live Recovery

Issue/Introduction

To fix No space issue on SRM appliance

Symptoms:

• Storage Replication Adapter (SRA) show inaccessible/missing in the Site Recovery UI
• Error: "No space left on device" -- seen in SRM server logs.
• SRM server logs location --> /var/log/vmware/vmware-dr.log

Environment

VMware Site Recovery Manager 8.x

Cause

The issue occurs if partition at /dev/loop0 --> /opt/vmware/support/logs/srm/SRAs gets full:

root [ /home/admin ]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        5.9G     0  5.9G   0% /dev
tmpfs                           5.9G   24K  5.9G   1% /dev/shm
tmpfs                           5.9G  900K  5.9G   1% /run
tmpfs                           5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda4                        14G  3.7G  9.2G  29% /
tmpfs                           5.9G   64K  5.9G   1% /tmp
/dev/sda2                       240M   26M  202M  12% /boot
/dev/mapper/support_vg-support   17G  2.5G   14G  16% /opt/vmware/support
/dev/loop0                      2.3G  2.2G     0 100% /opt/vmware/support/logs/srm/SRAs
overlay                          14G  3.7G  9.2G  29% /var/lib/docker/overlay2/b765a3bb515b74b9b22865658ddc2ff76d1c58c44b5d94891e73f3aca9e500bd/merged
overlay                          14G  3.7G  9.2G  29% /var/lib/docker/overlay2/528a19746188544ca8f8b89466682075567e63caecc320d7a8b481d3e319cc20/merged
tmpfs                           1.2G     0  1.2G   0% /run/user/666

 

 

Resolution

1. Take an offline snapshot of the SRM appliance
2. Add 2GB space each to both the Hard Disks of the SRM appliance
3. Power On the Appliance and follow the below steps to increase the partition size

root [ /home/admin ]# sudo losetup /dev/loop0
/dev/loop0: [65024]:49163 (/opt/vmware/support/logs/srm/.SRAs.img)
root [ /home/admin ]# sudo dd if=/dev/zero bs=1MiB of=/opt/vmware/support/logs/srm/.SRAs.img conv=notrunc oflag=append count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 2.92413 s, 717 MB/s
root [ /home/admin ]# sudo losetup -c /dev/loop0
root [ /home/admin ]# sudo resize2fs /dev/loop0
resize2fs 1.45.5 (07-Jan-2020)
Filesystem at /dev/loop0 is mounted on /opt/vmware/support/logs/srm/SRAs; on-line resizing required
old_desc_blocks = 10, new_desc_blocks = 18
The filesystem on /dev/loop0 is now 4505600 (1k) blocks long.

root [ /home/admin ]# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        5.9G     0  5.9G   0% /dev
tmpfs                           5.9G   24K  5.9G   1% /dev/shm
tmpfs                           5.9G  976K  5.9G   1% /run
tmpfs                           5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda4                        14G  3.7G  9.1G  29% /
tmpfs                           5.9G   64K  5.9G   1% /tmp
/dev/sda2                       240M   26M  202M  12% /boot
/dev/mapper/support_vg-support   17G  4.4G   12G  28% /opt/vmware/support
/dev/loop0                      4.2G  2.1G  2.0G  51% /opt/vmware/support/logs/srm/SRAs
overlay                          14G  3.7G  9.1G  29% /var/lib/docker/overlay2/cc0b5c00377b56a3f6f1b40840340098e2aeba4cd5c6cf69791c39b1baa6663f/merged
overlay                          14G  3.7G  9.1G  29% /var/lib/docker/overlay2/39261a3bd601c2141fc0cd76d590f4c1e34276e14b1ba75b5dcdfa3b1218d868/merged
overlay                          14G  3.7G  9.1G  29% /var/lib/docker/overlay2/9ce59c3de4286f02092192d7d5198550834af4a307a361dfd63820cea5afd677/merged
tmpfs                           1.2G     0  1.2G   0% /run/user/666

4. Rescan the Storage Replication Adapters for both sites in the DR Client and validate the Adapters are Visible and Healthy.
5. If the above succeeds, run a Device Discovery using the DR Client