Unable to use vCenter due to space issue. Found core.in:imuxsock.xxxxx core dump files

Products

• VMware vCenter Server

Issue/Introduction

Unable to use vCenter due to space running out.  Found core.in:imuxsock.xxxxx core dump files.

 

Environment

vSphere 7.0U3

Cause

When searching journalctl and find "too many open files" that type of issue can cause this type of coredump file.  The likely cause is that the number of logs in /var/log/vmware folder is more than the available inodes.

Search for number of logs using these commands:

 

find var/log/vmware/applmgmt/ -iname "*.stdout-*" | wc -l
find var/log/vmware/applmgmt/ -iname "*.stderr-*" | wc -l
ls var/log/vmware/vsphere-ui/logs/ | wc -l
find var/log/vmware/ -iname "messages.*.backup" | wc -l

 

If these numbers for the counts are in the thousands or higher then it is likely the high numbers of logs cause the problems.

Resolution

Before making changes to vCenter, it is a good idea to take a snapshot in case something gets deleted that might be essential.

 

Delete files using a carefully and targeted rm command.  

Example:

 

cd /storage/core

rm core.in:imuxsock.*

 

Alternatively you can use a more nuanced approach:

 

Stop all the services before next step
service-control --stop --all
wait for this to finish
service-control --status
make sure all the services are down.

Now delete the std{out|err} files.
find /var/log/vmware/ -type f -mtime +7 -iname "*.stdout-*" | xargs -i rm -f {}
find /var/log/vmware/ -type f -mtime +7 -iname "*.stderr-*" | xargs -i rm -f {}

Now delete all the rsyslog coredumps
find /storage/core/ -iname "core.in:imuxsock*" | xargs -i rm -f {}
find /storage/core/ -iname "core.in:imfile*" | xargs -i rm -f {}

restart the rsyslog service
systemctl restart rsyslog

Bring the services back
service-control --start --all

Wait for this to finish
service-control --status
make sure all the relevant services are up

Additional Information

Similar problems exist with imfile and these imuxsock files.

vCenter KB on imfiles and imuxsock files:

https://knowledge.broadcom.com/external/article?articleNumber=318159

 

SDDC manager imfile KB:

 

https://knowledge.broadcom.com/external/article?articleNumber=316053