vRealize Operations Application Remote Collector root filesystem is full

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• The/var/log partition usage increases with a particular log file 'cron', leading to bloating of the root filesystem, eventually running out of space and affecting the functionality of the Application Remote Collector appliance.

Environment

VMware vRealize Operations 8.x

Cause

This is caused by the cron log not being properly configured with a log rotator.

Resolution

This is a known issue affecting vRealize Operations 8.0 - 8.3, specifically with the Application Remote Collector.

Workaround:

 

For an Application Remote Collector with internet access

• SSH to the Application Remote Collector as root
• Install the logrotate package
  tdnf install logrotate

• Using a text editor, create a file cron-log in this location: /etc/logrotate.d/cron-log, with the content below:
  /var/log/cron {

                daily
                missingok
                copytruncate
                rotate 2
                size 1M
                compress
                notifempty
             }

• Take a backup of this file:
  cp /ucp/ucp-config-scripts/cron-input /ucp/ucp-config-scripts/cron-input_backup

• Using a text editor, append the following entry to the end of the original /ucp/ucp-config-scripts/cron-input file.  This will create a cron job entry to enable the log rotation for /var/log/cron which will run at 22:00 each day:
  0 22 * * * /usr/sbin/logrotate /etc/logrotate.d/cron-log

• Save your changes and exit the text editor
• Run the following command to submit this file to crontab
  crontab /ucp/ucp-config-scripts/cron-input

• To confirm it was added, run the following command. You should see the cron entry we added above in the output.
  crontab -l

 

For an Application Remote Collector without internet access

• Download the logrotate-3.9.1-2.ph1.x86_64.rpm from the attachments
• Copy the file to the /tmp directory of your Application Remote Collector Appliance
• SSH to the Application Remote Collector as root
• Install the logrotate package
  rpm -ivh logrotate-3.9.1-2.ph1.x86_64.rpm

• Using a text editor, create a file cron-log in this location: /etc/logrotate.d/cron-log, with the content below:
  /var/log/cron {
                  daily
                  missingok
                  copytruncate
                  rotate 2
                  size 1M
                  compress
                  notifempty
               }
• Take a backup of this file:
  cp /ucp/ucp-config-scripts/cron-input /ucp/ucp-config-scripts/cron-input_backup
• Using a text editor, append the following entry to the end of the original /ucp/ucp-config-scripts/cron-input file.  This will create a cron job entry to enable the log rotation for /var/log/cron which will run at 22:00 each day:
  0 22 * * * /usr/sbin/logrotate /etc/logrotate.d/cron-log
• Save your changes and exit the text editor
• Run the following command to submit this file to crontab
  crontab /ucp/ucp-config-scripts/cron-input
• To confirm it was added, run the following command. You should see the cron entry we added above in the output.
  crontab -l

Additional Information

Using the Application Remote Collector Migration Script in vRealize Operations 8.4 and above (315986)

Impact/Risks:
Please take a snapshot of your Application Remote Collector prior to performing these changes, and remember to remove them within ~72 hours if the change is successful.