SRM services may stop due to unmanaged SRA log file accumulation

Products

VMware Live Recovery

Issue/Introduction

Unmanaged SRM or SRA logs within the appliance can fill up support partition disk. This leads to the failure of the SRM server service or the inability to discover storage adapters because there is no remaining space to write metadata or logs.

Symptoms:

The srm-server service stops unexpectedly.
The site recovery client displays the below error messages under Storage Replication Adapter section.
"Unable to find SRA at the paired site"
"Unable to find SRA with UUID"
/opt/vmware/support/logs/srm/vmware-dr.log reports 'No space left on device' events.
-> Panic: Scheduled function triggered exception:
--> NSt10filesystem7__cxx1116filesystem_errorE filesystem error: cannot create directory: No space left on device [/var/log/vmware/srm/SRAs/sha256_#####################################]
--> Backtrace:
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 9.0.2, build: build-24401761, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[00] libvmacore.so[0x00252482]: Vmacore::System::Stacktrace::CaptureFullWork(unsigned int)
--> backtrace[01] libvmacore.so[0x0022B207]: Vmacore::System::SystemFactory::CreateBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] libvmacore.so[0x0048CA7B]
--> backtrace[03] libvmacore.so[0x0048CB82]: Vmacore::PanicExit(char const*)
--> backtrace[04] libdr-storage.so[0x000EB55B]
--> backtrace[05] libdr-storage.so[0x0014CB40]
--> backtrace[06] libvmacore.so[0x003429CE]
--> backtrace[07] libvmacore.so[0x003442D2]
--> backtrace[08] libvmacore.so[0x00497DE0]
--> backtrace[09] libpthread.so.0[0x00008EB0]
--> backtrace[10] libc.so.6[0x000FFADF]
--> backtrace[11] (no module)
--> [backtrace end]
The SRA partition shows 100% utilization
df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 4.9G 44K 4.9G 1% /dev/shm
tmpfs 2.0G 856K 2.0G 1% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/sda4 14G 5.5G 7.4G 43% /
tmpfs 4.9G 488K 4.9G 1% /tmp
/dev/sda2 238M 36M 190M 16% /boot
/dev/mapper/support_vg-support 3.9G 555M 3.2G 15% /opt/vmware/support
/dev/loop0 378M 358M 0 100% /opt/vmware/support/logs/srm/SRAs
Log bundles cannot be generated.
In some scenarios, the site recovery client reports below error messages.
Unable to retrieve pairs from extension server at https://####srm01.##.##.###:443/drserver/vcdr/vmomi/sdk.Unable to connect to Site Recovery Manager Server at https://###srm01.####.###.####:443/drserver/vcdr/vmomi/sdk. Reason: Unexpected status code: 503
Planned migration or test recovery fails with the following error message

"Failed to create snapshots of replica devices. SRA command 'testFailoverStart' failed. Internal error: Unhandled exception. System.IO.IOException: No space left on device : '/srm/sra/log/testFailoverStart
This issue may also occur when trying to add new or another SRA adapter to the existing SRM setup and the new SRA adapter fails to show.

Environment

VMware Site Recovery Manager (all versions)
VMware Live Site Recovery (all versions)

Cause

The SRM appliance has two main disks: system and support.

The support disk is partitioned into lvm ext4 filesystem & is used to store the logs of VMware-related services. The mount point of the partition is "/opt/vmware/support"

Some SRAs (e.g., PowerFlex, Dell) lack an inbuilt log rotation & compression software or don't have a limit on the size of logs they produce. This will result in the unplanned failure of the SRM server services due to the lack of any space available on disk for the log files. This may also result in a build up of 0-byte files

Resolution

This is a workaround and not a permanent fix.

VMware uses a logrotate Linux package that provides a way to clean up existing files without removing them. The required tools will be installed on the appliance and is attached to this KB known as clean-sras-logs.sh

What will the script do ?

This script will perform a manual clean up. After log rotation, the old logs will be stored in an archive until the next log rotation. The archive will be stored under "/opt/vmware/support/logs/srm/" folder.

First, the script will identify all of the subfolders that contain the SRAs logs & then create a configuration for logrotate to process these logs. Logrotate is then going to copy the logs to a folder outside of the SRA partition and truncate the original files without removing them.After that, all logs will be archived and stored on the main support partition.

How to install the script ?

Download the script clean-sras-logs.sh at the bottom of the article. Rename the file back to clean-sras-logs.sh once it is downloaded
Using WinSCP or equivalent application connect to the SRM appliance with the admin login account and password.
Copy the downloaded file into the "/home/admin" folder
Close the WinSCP or equivalent application
Connect to the SRM appliance with an SSH application like putty and login as admin then su as root user
Copy the place the script from the "/home/admin" folder to the "/opt/vmware/bin" location with the name clean-sras-logs.sh
Change permissions to make the script executable.
chmod 755 /opt/vmware/bin/clean-sras-logs.sh
For VLSR version post 9.0.3, manually create directory (mkdir -p /opt/vmware/srm/conf) prior running the script :
/opt/vmware/bin/clean-sras-logs.sh

How to enable cron job to execute SRAs log rotation everyday ?

Login to the appliance as admin and su as root user, create an empty file inside "/etc/cron.d/" with the filename sras.cron:

touch /etc/cron.d/sras.cron
Fill the content with the below text using a text editor :
0 0 * * * root /bin/bash /opt/vmware/bin/clean-sras-logs.shRefer ‘crontab’ in Linux with Examples to understand how to set these commands and work with them.
Restart the crond service:

systemctl restart crond

How to disable cron job to execute SRAs log rotation everyday ?

Login to the appliance as admin and su as root user, remove the file /etc/cron.d/sras.cron:

cd /etc/cron.d
rm sras.cron
Restart the crond service:

systemctl restart crond

How to remove the script ?

Login to the appliance as admin and su as root user, remove the file /opt/vmware/bin/clean-sras-logs.sh

cd /opt/vmware/bin rm clean-sras-logs.sh

Additional Information

Q. What is the default log rotation interval in SRM appliance and where is it located (path) ?
A. All log settings can be found here - /opt/vmware/srm/conf/vmware-dr.xml

If files are not being removed during the truncate due to SRA settings edit the "Template" section of the script from:

TEMPLATE='{
   su root root
   copytruncate
   olddir /opt/vmware/support/logs/srm/olddir
   notifempty
   missingok
   nocompress
   sharedscripts
   postrotate
      rm -f /opt/vmware/support/logs/srm/SRAs.tar.gz
      tar -cvzf /opt/vmware/support/logs/srm/SRAs.tar.gz /opt/vmware/support/logs/srm/olddir
      rm -rf /opt/vmware/support/logs/srm/olddir/*
   endscript
}'

To:

TEMPLATE='{
   su root root
   copytruncate
   olddir /opt/vmware/support/logs/srm/olddir
   nocreate
   notifempty
   missingok
   nocompress
   sharedscripts
   postrotate
      rm -f /opt/vmware/support/logs/srm/SRAs.tar.gz
      tar -cvzf /opt/vmware/support/logs/srm/SRAs.tar.gz /opt/vmware/support/logs/srm/olddir
      rm -rf /opt/vmware/support/logs/srm/olddir/*
   endscript
}'

Attachments

clean-sras-logs get_app