core.in:imuxsock.##### core dump files filling up /dev/mapper/core_vg-core mounted on /storage/core in vCenter

Products

• VMware vCenter Server

Issue/Introduction

• Unable to use vCenter due to space running out.
• Essential vCenter services might abruptly stop and fail to start.
• vCenter Storage status 'Critical' on VAMI

Environment

• VMware vSphere 7.0U3
• VMware vCenter Server Appliance 7.0.x
• VMware vCenter Server Appliance 8.0.x

Cause

1. vCenter Server Appliances' /storage/core partition filled up to 100% due to the generation of core dump files 'core.in:imuxsock.#####' 
2. When searching journalctl and find "too many open files" that type of issue can cause this type of coredump file.  The likely cause is that the number of logs in /var/log/vmware folder is more than the available inodes.
3. Search for number of logs using these commands:

   find /var/log/vmware/applmgmt/ -iname "*.stdout-*" | wc -l
   find /var/log/vmware/applmgmt/ -iname "*.stderr-*" | wc -l
   ls /var/log/vmware/vsphere-ui/logs/ | wc -l
   find /var/log/vmware/ -iname "messages.*.backup" | wc -l

   
   
   
4. If these numbers for the counts are in the thousands or higher then it is likely the high numbers of logs cause the problems.

 

Resolution

Note: Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 

 

1. SSH to vCenter as Root
   
   
2. Delete files using a carefully and targeted rm command.  
   
   

   cd /storage/core 
   rm core.in:imuxsock.*

   
   
    
3. If the core.in:imuxsock.xxxxx files continue to generate, remove the logs that are in the thousands or more from the below directories  
   
   

   Cleaned-up all the `.backup` files and `.std{out|err}-` files : 
   
   find /var/log/vmware -type f -name "*.backup" | xargs -i rm -f {}
   find /var/log -type f -name "lastlog.*.backup" | xargs -i rm -f {}
   find /var/log -type f -name "tallylog.*.backup" | xargs -i rm -f {}

   
   
   
4. Stop all the services before next step
   
   

   service-control --stop --all

    

5. Make sure all the services are down by checking the service status.
   
   

   service-control --status
   

    

6. Delete the std{out|err} files. 
   
   

   find /var/log/vmware/ -type f -mtime +7 -iname "*.stdout-*" | xargs -i rm -f {} find /var/log/vmware/ -type f -mtime +7 -iname "*.stderr-*" | xargs -i rm -f {}

    

7. Delete all the rsyslog core dumps
   
   

   find /storage/core/ -iname "core.in:imuxsock*" | xargs -i rm -f {}
   find /storage/core/ -iname "core.in:imfile*" | xargs -i rm -f {}
   

    
8. Delete the vsphere-ui files
   
   

   cd /var/log/vmware/vsphere-ui/logs
   rm -f vsphere-ui-gc.log.*

    
9. Delete the backup messages files
   
   

   cd /var/log/vmware/
   rm -f messages.*.backup

    

10. Restart the rsyslog service
    
    

    systemctl restart rsyslog

     

11. Bring the services back
    
    

    service-control --start --all

     

12. Wait for this to finish
    
    

    service-control --status

     

13. Make sure all the relevant services are up

Additional Information

Similar problems exist with imfile and these imuxsock files.

vCenter KB on imfiles and imuxsock files:

vCenter Server /storage/core filling up with core.in:imfile.XXXX or core.in:imuxsock.XXXX when configuring vCenter syslog server

SDDC manager imfile KB:

SDDC Manager appliance root file system is filled with files (imfile-state:***)

 

The issue can also cause an upgrade failure on vCenter 7.x (example - upgrading from 7.0u3t to 7.0u3v).

The VAMI will display these errors -

• Installation failed
• Exception occured in postInstallHook
• There is not enough space on the disk(s). Increase disk space and try again.

Implementing the resolution above, and then retrying the upgrade will allow it to succeed.