SRA log partition full in SRM appliance

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Storage Replication Adapter (SRA) is missing in the Site Recovery UI

• The SRA log partition is full in the SRM appliance. 

  root@srm [ ~ ]# df -h

  Filesystem                      Size  Used Avail Use% Mounted on

  devtmpfs                        4.0M     0  4.0M   0% /dev

  tmpfs                           4.9G   44K  4.9G   1% /dev/shm

  tmpfs                           2.0G  856K  2.0G   1% /run

  tmpfs                           4.0M     0  4.0M   0% /sys/fs/cgroup

  /dev/sda4                        14G  5.5G  7.4G  43% /

  tmpfs                           4.9G  488K  4.9G   1% /tmp

  /dev/sda2                       238M   36M  190M  16% /boot

  /dev/mapper/support_vg-support  3.9G  555M  3.2G  15% /opt/vmware/support

  /dev/loop0                      378M  358M     0 100% /opt/vmware/support/logs/srm/SRAs

• In the SRM /var/log/vmware/vmware-dr.log, the following messages are logged, indicating that SRA commands are failing due to "No space left on device."

2025-02-27T13:59:07.257Z error vmware-dr[01269] [SRM@6876 sub=SraCommand opID=c47d7f92] Command discoverArrays's stderr:
--> log4j:ERROR Failed to flush writer,
--> java.io.IOException: No space left on device
-->     at java.io.FileOutputStream.writeBytes(Native Method)
-->     at java.io.FileOutputStream.write(FileOutputStream.java:326)
-->     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
-->     at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
-->     at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
-->     at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
-->     at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
-->     at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:57)
-->     at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:315)

 

Environment

VMware Site Recovery Manager 8.x
VMware Live Site Recovery 9.x

Cause

The SRM appliance has two main disks: system and support.

The support disk is partitioned into lvm ext4 filesystem & is used to store the logs of VMware-related services. The mount point of the partition is "/opt/vmware/support"

Some SRAs (Storage Replication Adapter) lack an inbuilt log rotation & compression software or don't have a limit on the size of logs they produce. This will result in the unplanned failure of the SRM server services due to the lack of any space available on disk for the log files.

Resolution

This is a workaround and not a permanent fix. 

VMware uses a logrotate Linux package that provides a way to clean up existing files without removing them. The required tools will be installed on the appliance and is attached to this KB known as clean-sras-logs.sh

What will the script do ? 

This script will perform a manual clean up. After log rotation, the old logs will be stored in an archive until the next log rotation. The archive will be stored under "/opt/vmware/support/logs/srm/" folder.

First, the script will identify all of the subfolders that contain the SRAs logs & then create a configuration for logrotate to process these logs. Logrotate is then going to copy the logs to a folder outside of the SRAs partition and truncate the original files without removing them.After that, all logs will be archived and stored on the main support partition.

How to install the script ?

1. Download the script clean-sras-logs.sh at the bottom of the article. 
2. Login to the appliance as admin and su as root user, place the script in "/opt/vmware/bin" location with the name clean-sras-logs.sh
3. Change permissions to make the script executable.  

chmod 755 /opt/vmware/bin/clean-sras-logs.sh

4. Manually execute the script to rotate the log files inside the SRAs log partition.

/opt/vmware/bin/clean-sras-logs.sh

How to enable cron job to execute SRAs log rotation everyday ? 

1. Login to the appliance as admin and su as root user, create an empty file inside "/etc/cron.d/" with the filename sras.cron:

touch /etc/cron.d/sras.cron

2. Fill the content with the below text using a text editor :

0 0 * * * root /bin/bash /opt/vmware/bin/clean-sras-logs.sh

You can check out ‘crontab’ in Linux with Examples to understand how to set these commands and work with them. 

3. Restart the crond service:

systemctl restart crond

How to disable cron job to execute SRAs log rotation everyday ? 

1. Login to the appliance as admin and su as root user, remove the file /etc/cron.d/sras.cron:

cd /etc/cron.d
rm sras.cron

2. Restart the crond service:

systemctl restart crond

How to remove the script ? 

1. Login to the appliance as admin and su as root user, remove the file /opt/vmware/bin/clean-sras-logs.sh

cd /opt/vmware/bin
rm clean-sras-logs.sh