vRealize Operations Application Remote Collector root filesystem is full

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The/var/log partition usage increases with a particular log file 'cron', leading to bloating of the root filesystem, eventually running out of space and affecting the functionality of the Application Remote Collector appliance

Environment

VMware vRealize Operations 8.x

Cause

This is caused by the cron log not being properly configured with a log rotator

Resolution

This is a known issue affecting vRealize Operations 8.0-8.3, specifically with the Application Remote Collector.  The currently recommended solution is to migrate to 

Workaround:
To workaround the issue:

For an Application Remote Collector with external internet access

1. SSH to the Application Remote Collector as root
2. Install the logrotate package

tdnf install logrotate

3. Using a text editor, create a file cron-log in this location: /etc/logrotate.d/cron-log, with the beneath content

/var/log/cron {
                daily
                missingok
                copytruncate
                rotate 2
                size 1M
                compress
                notifempty
             }

4. Take a backup of this file:

cp /ucp/ucp-config-scripts/cron-input /ucp/ucp-config-scripts/cron-input_backup

5. Using a text editor, append the following entry to the end of the original /ucp/ucp-config-scripts/cron-input file.  This will create a cron job entry to enable the log rotation for /var/log/cron which will run at 22:00 each day:

0 22 * * * /usr/sbin/logrotate /etc/logrotate.d/cron-log

5. Save your changes and exit the text editor
6. Run the following command to submit this file to crontab

crontab /ucp/ucp-config-scripts/cron-input

8. To confirm it was added, run the following command. You should see the cron entry we added in step 5 in the output.

crontab -l

For an Application Remote Collector without external internet access

1. Download the logrotate-3.9.1-2.ph1.x86_64.rpm from the attachments
2. Copy the file to the /tmp directory of your Application Remote Collector Appliance
3. SSH to the Application Remote Collector as root
4. Install the logrotate package

rpm -ivh logrotate-3.9.1-2.ph1.x86_64.rpm

3. Using a text editor, create a file cron-log in this location: /etc/logrotate.d/cron-log, with the beneath content

/var/log/cron {
                daily
                missingok
                copytruncate
                rotate 2
                size 1M
                compress
                notifempty
             }

4. Take a backup of this file:

cp /ucp/ucp-config-scripts/cron-input /ucp/ucp-config-scripts/cron-input_backup

5. Using a text editor, append the following entry to the end of the original /ucp/ucp-config-scripts/cron-input file.  This will create a cron job entry to enable the log rotation for /var/log/cron which will run at 22:00 each day:

0 22 * * * /usr/sbin/logrotate /etc/logrotate.d/cron-log

5. Save your changes and exit the text editor
6. Run the following command to submit this file to crontab

crontab /ucp/ucp-config-scripts/cron-input

8. To confirm it was added, run the following command. You should see the cron entry we added in step 5 in the output.

crontab -l

Additional Information

Using the Application Remote Collector Migration Script in vRealize Operations 8.4 and above (83059)

Impact/Risks:
Please take a snapshot of your Application Remote Collector prior to performing these changes, and remember to remove them after ~72 hours if the change is successful.