vCenter Server's /storage/core partitiion is filling up with core.in:imfile.#### or core.in:imuxsock.#### when configuring syslog server

Products

• VMware vCenter Server

Issue/Introduction

• After configuring the vCenter with a remote syslog server, the following error messages appear in the /var/log/vmware/rsyslogd/rsyslogd-syslog.log

err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-YYYYMMDDHHMMSS': Too many open files v8.2306.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-YYYYMMDDHHMMSS': Too many open files v8.2306.0]

• Additionally, multiple core files such as core.in:imfile.#### or core.in:imuxsock.#### are found under vCenter's /storage/core/ partition.

total 50738008
-r-x-  1 root   root  218468352 MMM DD HH:MM core.in: imuxsock.10087
-r-x-  1 root   root  218468232 MMM DD HH:MM core.in: imuxsock.1017
-r-x-  1 root   root  218468452 MMM DD HH:MM core.in: imuxsock.10195
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.10475
-r-x-  1 root   root  218468367 MMM DD HH:MM core.in: imuxsock.10833
-r-x-  1 root   root  218468334 MMM DD HH:MM core.in: imuxsock.11547
-r-x-  1 root   root  218468378 MMM DD HH:MM core.in: imuxsock.11632
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.11648

Environment

• VMware vCenter Server 7.x

Cause

The issue occurs due to a file descriptor (FD) limit being exceeded by the rsyslog service, especially when monitoring a large number of files.

Resolution

The issue has been resolved in vSphere version 7.0 Update 3q and later. 

If upgrading or patching the vCenter Server to a build containing the fix is not feasible, the following workaround can be applied.

Note: Ensure there is a valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

Workarounds

Workaround 1: Reconfigure the postgres-archiver rsyslog input

1. Back up the existing configuration
   
   cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig


2. Edit the configuration file
   
   vi /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf
   
   
3. Replace the contents with the following:
   
   #vmware-postgres-archiver logs
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
   
   
4. Restart rsyslog
   
   systemctl restart rsyslog
   
   
5. Verify the service status:
   
   systemctl status rsyslog

syslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri YYYY-MM-DD HH:MM:SS UTC; SSs ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 38846 (rsyslogd)
Tasks: ## (limit: 9830)
Memory: ##M
CGroup: /system.slice/rsyslog.service
└─38846 /usr/sbin/rsyslogd -n

Workaround 2: Remove core files and schedule cleanup

1. Stop rsyslog
   
   systemctl stop rsyslog
   
   
2. Delete existing core files
   
rm -f /storage/core/core.in:imfile*
rm -f /storage/core/core.in:imuxsock*
   
   
3. Clean up old access logs:
   
   find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
   
   
4. (Optional) Automate cleanup with a cron job
   1. Create a script:
      
vi /root/cleanupAccessLog.sh
      
      
   2. Add the following:
      
      #!/bin/bash
find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
find /storage/core/ -mtime +1 -type f -name "core.in:imfile*" | while read file; do rm "$file"; done
find /storage/core/ -mtime +1 -type f -name "core.in:imuxsock*" | while read file; do rm "$file"; done
      
      
   3. Make the script executable
      
chmod 755 /root/cleanupAccessLog.sh
      
      
   4. Schedule with cron
      
      vi /etc/cron.d/cleanAccessLog.cron


   5. Add:
      
0 0 * * 0 root /root/cleanupAccessLog.sh 2>&1
      
      
   6. Restart rsyslog
      
      systemctl start rsyslog
      

If the issue still persists after following the above steps, follow the resolution steps from KB core.in:imuxsock.xxxxx core dump files filling up /dev/mapper/core_vg-core mounted on /storage/core in vCenter.