/storage/log filling up with imfile-state files | rsyslogd

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

/storage/log is filling up and there are many files in the /var/log/vmware/rsyslogd directory.  The contents of /var/log/vmware/rsyslogd/ may or may not be listed.

Environment

VMware vCenter Server 7.0.x

 

VMware vCenter Server 8.0.x

Cause

The imfile-state:XXXX files are created due to the rsyslog daemon's method of tracking messages that have already been sent to the syslog server.

Resolution

This issue is resolved in VMware vCenter Server 7.0 Update 3q (P09) Build 23788036

To workaround this issue, create a cron job manually to clear the files.

1. Login to vCenter Server using ssh

2. Remove all the files immediately (before creating the cron job), run the following command and wait until it is finished
   
   find /var/log/vmware/rsyslogd/ -name "imfile-state*" | xargs -i rm -f {}

3. Stop rsyslog
   
   systemctl stop syslog.socket rsyslog.service
   
   
4. Verify that the service is stopped
   
   ps aux | grep rsyslogd
   
   
5. Verify in /var/run that rsyslogd.pid does not exist
   
   
6. Create a directory for the cron job
   
   cd ~
mkdir syslogcron
cd syslogcron/
   
   
7. Create a file called cleanimf.sh and save it with contents below.
   
   vi cleanimf.sh
   
   #!/bin/bash
   find /var/log/vmware/rsyslogd/ -name "imfile-state*" | xargs -i rm -f {}
   
   
8. After saving the file (by typing ':wq'), apply permissions to the file
   
   chmod 755 cleanimf.sh
   
   
9. Create a cron job to execute the script every hour (remember to save the file with :wq)
   
   vi /etc/cron.d/cleanimf.cron
   
   0 * * * * root /root/syslogcron/cleanimf.sh 2>&1
   
   
10. Start the rsyslog daemon
    
    systemctl start syslog.socket rsyslog.service

Additional Information

The time interval that the cron job removes these files may need to be increased to remove the files at the rate they are created

We can alter the cron job to run it every 30 minutes instead of every hour by altering step 9 with the command bellow:

Change 

"0 * * * * root /root/syslogcron/cleanimf.sh 2>&1" 

To

"*/30 * * * * root /root/syslogcron/cleanimf.sh 2>&1"