vCenter Server /storage/core filling up with core.in:imfile.XXXX or core.in:imuxsock.XXXX when configuring vCenter syslog server

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

After configuring a remote syslog server, the following error messages appear in the rsyslogd-syslog.log
 
<YYYY-MM-DD><time>Z err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
<YYYY-MM-DD><time>Z err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]

Additionally, multiple core files such as core.in:imfile.XXXX or core.in:imuxsock.XXXX are found under /storage/core.

total 50738008
-r-x-  1 root   root  218468352 MMM DD HH:MM core.in: imuxsock.10087
-r-x-  1 root   root  218468232 MMM DD HH:MM core.in: imuxsock.1017
-r-x-  1 root   root  218468452 MMM DD HH:MM core.in: imuxsock.10195
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.10475
-r-x-  1 root   root  218468367 MMM DD HH:MM core.in: imuxsock.10833
-r-x-  1 root   root  218468334 MMM DD HH:MM core.in: imuxsock.11547
-r-x-  1 root   root  218468378 MMM DD HH:MM core.in: imuxsock.11632
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.11648

 Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

 

Environment

VMware vCenter Server 7.0.x

Cause

The issue occurs due to a file descriptor (FD) limit being exceeded by the rsyslog service, especially when monitoring a large number of files.

Resolution

This is a known issue in vCenter Server 7.0.

To permanently fix the problem, upgrade to vCenter Server 7.0 U3l or newer.

To download the latest version of vCenter Server Appliance (vCSA), refer to the My VMware Downloads portal My Downloads .

Workarounds:
Please follow any one of the below workarounds :

Workaround 1: Reconfigure the postgres-archiver rsyslog input

 

1. Backup the existing configuration:
   #cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig
    

2. Edit the configuration file:
   #vi /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf
   
   

3. Replace the contents with the following:

   # vmware-postgres-archiver logs

   input(type="imfile"

   File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"

   Tag="postgres-archiver"

   Severity="info"

   Facility="local0")

   input(type="imfile"

   File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"

   Tag="postgres-archiver"

   Severity="info"

   Facility="local0")
   
   

4. Restart rsyslog:
   #systemctl restart rsyslog
   
   

5. Verify the service status:
   #systemctl status rsyslog

   syslog.service - System Logging Service

   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)

   Active: active (running) since Fri YYYY-MM-DD HH:MM:SS UTC; 32s ago

   Docs: man:rsyslogd(8)

   http://www.rsyslog.com/doc/

   Main PID: 38846 (rsyslogd)

   Tasks: 13 (limit: 9830)

   Memory: 5.3M

   CGroup: /system.slice/rsyslog.service

   └─38846 /usr/sbin/rsyslogd -n

    

 

Workaround 2: Remove core files and schedule cleanup
 

1. Stop rsyslog:
   # systemctl stop rsyslog

2. Delete existing core files:

   #rm -f /storage/core/core.in:imfile*

   #rm -f /storage/core/core.in:imuxsock*
   
   

3. Clean up old access logs:
   #find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
   
   

4. (Optional) Automate cleanup with a cron job:
   
   

   Create a script:
   #vi /root/cleanupAccessLog.sh
   
   

   Add the following:

   #!/bin/bash

   find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done

   find /storage/core/ -mtime +1 -type f -name "core.in:imfile*" | while read file; do rm "$file"; done

   find /storage/core/ -mtime +1 -type f -name "core.in:imuxsock*" | while read file; do rm "$file"; done

    

   Make the script executable:
   #chmod 755 /root/cleanupAccessLog.sh
    

   Schedule with cron:
   vi /etc/cron.d/cleanAccessLog.cron
    

   Add:
   0 0 * * 0 root /root/cleanupAccessLog.sh 2>&1
   
   

5. Restart rsyslog:
   # systemctl start rsyslog