vCenter Server's /storage/core partition is filling up with core.in:imfile.#### or core.in:imuxsock.#### when configuring syslog server

Products

• VMware vCenter Server

Issue/Introduction

• After configuring the vCenter with a remote syslog server, the following error messages appear in the /var/log/vmware/rsyslogd/rsyslogd-syslog.log

err rsyslogd imfile error trying to access state file for '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2027 ]
err rsyslogd file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': open error: Too many open files [v8.2001.0 try https://www.rsyslog.com/e/2433 ]
err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/sso/tomcat/localhost_access.<YYYY-MM-DD>.log': Too many open files [v8.2001.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-YYYYMMDDHHMMSS': Too many open files v8.2306.0]
err rsyslogd imfile: error accessing file '/var/log/vmware/vapi/endpoint/vmware-vapi-endpoint. stdout-YYYYMMDDHHMMSS': Too many open files v8.2306.0]

• Additionally, multiple core files such as the following can be found under vCenter's /storage/core/ partition:
  • core.in:imfile.####
  • core.in:imuxsock.####
  • core.in:imptcp.####

total 50738008
-r-x-  1 root   root  218468352 MMM DD HH:MM core.in: imuxsock.10087
-r-x-  1 root   root  218468232 MMM DD HH:MM core.in: imuxsock.1017
-r-x-  1 root   root  218468452 MMM DD HH:MM core.in: imuxsock.10195
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.10475
-r-x-  1 root   root  218468367 MMM DD HH:MM core.in: imuxsock.10833
-r-x-  1 root   root  218468334 MMM DD HH:MM core.in: imuxsock.11547
-r-x-  1 root   root  218468378 MMM DD HH:MM core.in: imuxsock.11632
-r-x-  1 root   root  218468345 MMM DD HH:MM core.in: imuxsock.11648

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

The issue occurs due to a file descriptor (FD) limit being exceeded by the rsyslog service, especially when monitoring a large number of files.

Resolution

Please update to the latest version of vCenter Server. If upgrading is not feasible, the following workaround can be applied.

Note: Ensure there is a valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

Workarounds

1. Workaround 1: Reconfigure the postgres-archiver rsyslog input

   1. Back up the existing configuration
      
      cp /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf.orig


   2. Edit the configuration file
      
      vi /etc/vmware-syslog/vmware-services-vmware-postgres-archiver.conf
      
      
   3. Replace the contents with the following:
      
      #vmware-postgres-archiver logs
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stdout"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
input(type="imfile"
File="/var/log/vmware/vpostgres/pg_archiver.log.stderr"
Tag="postgres-archiver"
Severity="info"
Facility="local0")
      
      
   4. Restart rsyslog
      
      systemctl restart rsyslog
      
      
   5. Verify the service status:
      
      systemctl status rsyslog

syslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri YYYY-MM-DD HH:MM:SS UTC; SSs ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 38846 (rsyslogd)
Tasks: ## (limit: 9830)
Memory: ##M
CGroup: /system.slice/rsyslog.service
└─38846 /usr/sbin/rsyslogd -n

   Workaround 2: Remove core files and schedule cleanup

   1. Stop rsyslog
      
      systemctl stop rsyslog
      
      
   2. Delete existing core files
      
rm -f /storage/core/core.in:imfile*
rm -f /storage/core/core.in:imuxsock*
rm -f /storage/core/core.in:imptcp*
      
      
   3. Clean up old access logs:
      
      find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
      
      
   4. (Optional) Automate cleanup with a cron job
      1. Create a script:
         
vi /root/cleanupAccessLog.sh
         
         
      2. Add the following:
         
         #!/bin/bash
find /var/log/vmware/ -mtime +1 -type f -name "localhost*access*log*" | while read file; do rm "$file"; done
find /storage/core/ -mtime +1 -type f -name "core.in:imfile*" | while read file; do rm "$file"; done
find /storage/core/ -mtime +1 -type f -name "core.in:imuxsock*" | while read file; do rm "$file"; done
find /storage/core/ -mtime +1 -type f -name "core.in:imptcp*" | while read file; do rm "$file"; done
         
         
      3. Make the script executable
         
chmod 755 /root/cleanupAccessLog.sh
         
         
      4. Schedule with cron
         
         vi /etc/cron.d/cleanAccessLog.cron


      5. Add:
         
0 0 * * 0 root /root/cleanupAccessLog.sh 2>&1
         
         
      6. Restart rsyslog
         
         systemctl start rsyslog
         

   
   If the issue still persists after following the above steps, follow the resolution steps from KB core.in:imuxsock.xxxxx core dump files filling up /dev/mapper/core_vg-core mounted on /storage/core in vCenter.