vCenter Server /storage/core filling up with core.in:imfile.#### or core.in:imuxsock.#### when configuring vCenter syslog server
search cancel

vCenter Server /storage/core filling up with core.in:imfile.#### or core.in:imuxsock.#### when configuring vCenter syslog server

book

Article ID: 318159

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

After configuring a remote syslog server, the following error messages appear in the /var/log/vmware/rsyslogd/rsyslogd-syslog.log
 

<YYYY-MM-DD><time>Z err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
<YYYY-MM-DD><time>Z err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-20250726031001': Too many open files v8.2306.0]
<YYYY-MM-DD><time>Z err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-20250726031001': Too many open files v8.2306.0]


Additionally, multiple core files such as core.in:imfile.#### or core.in:imuxsock.#### are found under /storage/core/.

total 50738008
-r-x-  1 root   root  218468352 MMM DD HH:MM core.in: imuxsock.10087
-r-x-  1 root   root  218468232 MMM DD HH:MM core.in: imuxsock.1017
-r-x-  1 root   root  218468452 MMM DD HH:MM core.in: imuxsock.10195
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.10475
-r-x-  1 root   root  218468367 MMM DD HH:MM core.in: imuxsock.10833
-r-x-  1 root   root  218468334 MMM DD HH:MM core.in: imuxsock.11547
-r-x-  1 root   root  218468378 MMM DD HH:MM core.in: imuxsock.11632
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.11648

 Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

  • VMware vCenter Server 7.0.x
  • VMware vCenter Server 8.0.x

Cause

  • The issue occurs due to a file descriptor (FD) limit being exceeded by the rsyslog service, especially when monitoring a large number of files.

Resolution

This is a known issue in vCenter Server 7.0.

To permanently fix the problem, upgrade to vCenter Server 7.0 U3l or newer.

To download the latest version of vCenter Server Appliance (vCSA), refer to the My VMware Downloads portal My Downloads .

Note: Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 

 

Workarounds:
Please follow any one of the below workarounds :

Workaround 1: Reconfigure the postgres-archiver rsyslog input

  1. Backup the existing configuration:
    #cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig
  2. Edit the configuration file:
    #vi /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf
  3. Replace the contents with the following:

     # vmware-postgres-archiver logs

     input(type="imfile"

     File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"

     Tag="postgres-archiver"

     Severity="info"

     Facility="local0")

     input(type="imfile"

     File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"

     Tag="postgres-archiver"

     Severity="info"

     Facility="local0")

  4. Restart rsyslog:
    #systemctl restart rsyslog

  5. Verify the service status:
    #systemctl status rsyslog

     syslog.service - System Logging Service

     Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)

     Active: active (running) since Fri YYYY-MM-DD HH:MM:SS UTC; 32s ago

     Docs: man:rsyslogd(8)

     http://www.rsyslog.com/doc/

     Main PID: 38846 (rsyslogd)

     Tasks: 13 (limit: 9830)

     Memory: 5.3M

     CGroup: /system.slice/rsyslog.service

     └─38846 /usr/sbin/rsyslogd -n

Workaround 2: Remove core files and schedule cleanup
 

  1. Stop rsyslog:
    # systemctl stop rsyslog
  2. Delete existing core files:
     #rm -f /storage/core/core.in:imfile*

     #rm -f /storage/core/core.in:imuxsock*
  3. Clean up old access logs:
    #find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
  4. (Optional) Automate cleanup with a cron job:

     Create a script:
    #vi /root/cleanupAccessLog.sh

     Add the following:

     #!/bin/bash

     find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done

     find /storage/core/ -mtime +1 -type f -name "core.in:imfile*" | while read file; do rm "$file"; done

     find /storage/core/ -mtime +1 -type f -name "core.in:imuxsock*" | while read file; do rm "$file"; done

     Make the script executable:
    #chmod 755 /root/cleanupAccessLog.sh

    Schedule with cron:
    vi /etc/cron.d/cleanAccessLog.cron

    Add:
    0 0 * * 0 root /root/cleanupAccessLog.sh 2>&1


  5. Restart rsyslog:
    # systemctl start rsyslog


If the issue still persists, after following the above steps. Please follow the resolution steps from KB : core.in:imuxsock.xxxxx core dump files filling up /dev/mapper/core_vg-core mounted on /storage/core in vCenter

Powered by Wolken Software