vCenter Server /storage/core filling up with core.in:imfile.XXXX when configuring vCenter syslog server

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

After configuration of remote syslog server 

in rsyslogd-syslog.log you see messages similar to:

<YYYY-MM-DD><time>Z err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
<YYYY-MM-DD><time>Z err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 7.0.x

Cause

This is an issue due to FD overshoot of rsyslog.

Resolution

This is a known issue with vCenter 7.0. To fix this issue upgrade to vCenter 7.0 U3c and newer, to download latest version of vCSA 7.0 see VMware Download Center .

Workaround:
Please follow any one of the below workarounds :

Workaround 1 :

1. Take a backup of the original postgres-archiver syslog configuration:

cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig

2. Using vi or vim edit the file :

/etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

3. Remove the existing content and replace it with the below:

# vmware-postgres-archiver logs
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"
Tag="postgres-archiver"
Severity="info"
Facility="local0")

4. Restart rsyslog:

systemctl restart rsyslog

5. confirm that rsyslog is running:

systemctl status rsyslog

6. The output of the above command should be similar to the below:

rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-03-05 00:33:37 UTC; 32s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 38846 (rsyslogd)
Tasks: 13 (limit: 9830)
Memory: 5.3M
CGroup: /system.slice/rsyslog.service
└─38846 /usr/sbin/rsyslogd -n

Workaround 2 :

• Stop rsyslog using command "systemctl stop rsyslog"
• Delete the core files in /storage/core with pattern "core.in:imfile*"
• Then use the below workaround to avoid any further core dump issues.
• Run the below command :

         find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file" ; done
 

• Create a shell script with below content :   

         vi cleanupAccessLog.sh
         #!/bin/bash
         find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file" ; done
 

• Provide execute permission to the script :

   
           chmod 755 cleanupAccessLog.sh
 

• Navigate to cron job folder and create a cron file to execute the script on a regular basis :  

           cd /etc/cron.d
           vi cleanAccessLog.cron
           0 0 * * 0 root <path of the script>/cleanupAccessLog.sh 2>&1
 

• Post this start the Syslog service :

           systemctl start rsyslog