core.in:imuxsock.xxxxx core dump files filling up /dev/mapper/core_vg-core mounted on /storage/core in vCenter

Products

• VMware vCenter Server

Issue/Introduction

Unable to use vCenter due to space running out.  Found core.in:imuxsock.xxxxx core dump files.

Environment

vSphere 7.0U3

Cause

When searching journalctl and find "too many open files" that type of issue can cause this type of coredump file.  The likely cause is that the number of logs in /var/log/vmware folder is more than the available inodes.

Search for number of logs using these commands:

find /var/log/vmware/applmgmt/ -iname "*.stdout-*" | wc -l
find /var/log/vmware/applmgmt/ -iname "*.stderr-*" | wc -l
ls /var/log/vmware/vsphere-ui/logs/ | wc -l
find /var/log/vmware/ -iname "messages.*.backup" | wc -l

 

If these numbers for the counts are in the thousands or higher then it is likely the high numbers of logs cause the problems.

Resolution

• NOTE: *** Before removing any files take a powered off snapshot of all linked vCenters, powered off at the same time and ensure a file-based backup of the vCenter is taken. A non-linked vCenter can have a snapshot taken while running.  ***
• Backing up vCenter Server

 

1.Delete files using a carefully and targeted rm command.  

• Example:

• cd /storage/core 
  rm core.in:imuxsock.*

 

2.If the core.in:imuxsock.xxxxx files continue to generate, remove the logs that are in the thousands or more from the below directories 

/var/log/vmware/applmgmt/xxx.stdout
/var/log/vmware/applmgmt/xxx.stderr
/var/log/vmware/vsphere-ui/logs/vsphere-ui-gc.log.*
/var/log/vmware/messages.xxx.backup

3.Stop all the services before next step

service-control --stop --all

4.Wait for this to finish

service-control --status

5.Make sure all the services are down.

6.Delete the std{out|err} files.

find /var/log/vmware/ -type f -mtime +7 -iname "*.stdout-*" | xargs -i rm -f {}
find /var/log/vmware/ -type f -mtime +7 -iname "*.stderr-*" | xargs -i rm -f {}

7.Delete all the rsyslog coredumps

find /storage/core/ -iname "core.in:imuxsock*" | xargs -i rm -f {}
find /storage/core/ -iname "core.in:imfile*" | xargs -i rm -f {}


8.Delete the vsphere-ui files 

cd /var/log/vmware/vsphere-ui/logs
rm -f vsphere-ui-gc.log.*

9.Delete the backup messages files 

cd /var/log/vmware/
rm -f messages.*.backup

10.Restart the rsyslog service

systemctl restart rsyslog

11.Bring the services back

service-control --start --all

12.Wait for this to finish

service-control --status

13.Make sure all the relevant services are up

Additional Information

Similar problems exist with imfile and these imuxsock files.

vCenter KB on imfiles and imuxsock files:

vCenter Server /storage/core filling up with core.in:imfile.XXXX or core.in:imuxsock.XXXX when configuring vCenter syslog server

SDDC manager imfile KB:

SDDC Manager appliance root file system is filled with files (imfile-state:***)