vCenter Server /storage/core filling up with core.in:imfile.XXXX or core.in:imuxsock.XXXX when configuring vCenter syslog server

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

After configuration of remote syslog server in rsyslogd-syslog.log messages similar to below are found:

<YYYY-MM-DD><time>Z err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
<YYYY-MM-DD><time>Z err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]

In /storage/core, it has many file name core.in:imfile.XXXX or core.in:imuxsock.XXXX

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

 

Environment

VMware vCenter Server 7.0.x

Cause

This is an issue due to FD overshoot of rsyslog.

Resolution

This is a known issue with vCenter 7.0. To fix this issue upgrade to vCenter 7.0 U3l and newer, to download latest version of vCSA 7.0 see My Downloads .

Workaround:
Please follow any one of the below workarounds :

Workaround 1 :

1. Take a backup of the original postgres-archiver syslog configuration:

cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig

2. Using vi or vim edit the file :

/etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf

3. Remove the existing content and replace it with the below:

# vmware-postgres-archiver logs
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"
Tag="postgres-archiver"
Severity="info"
Facility="local0")

4. Restart rsyslog:

systemctl restart rsyslog

5. confirm that rsyslog is running:

systemctl status rsyslog

6. The output of the above command should be similar to the below:

rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2021-03-05 00:33:37 UTC; 32s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 38846 (rsyslogd)
Tasks: 13 (limit: 9830)
Memory: 5.3M
CGroup: /system.slice/rsyslog.service
└─38846 /usr/sbin/rsyslogd -n

Workaround 2 :

• Stop rsyslog using command "systemctl stop rsyslog"
• Delete the core files in /storage/core with pattern "core.in:imfile*" and "core.in:imuxsock*"
• Then use the below workaround to avoid any further core dump issues.
• Run the below command :

         find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file" ; done
 

• Create a shell script with below content :   

         vi cleanupAccessLog.sh
         #!/bin/bash
         find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file" ; done         

find /storage/core/ -mtime +1 -type f -name "core.in:imfile*" | while read file; do rm "$file" ; done

find /storage/core/ -mtime +1 -type f -name "core.in:imuxsock*" | while read file; do rm "$file" ; done

 

• Provide execute permission to the script :

   
           chmod 755 cleanupAccessLog.sh
 

• Navigate to cron job folder and create a cron file to execute the script on a regular basis :  

           cd /etc/cron.d
           vi cleanAccessLog.cron
           0 0 * * 0 root <path of the script>/cleanupAccessLog.sh 2>&1
 

• Post this start the Syslog service :

           systemctl start rsyslog