"core.rs:main Q:Reg.XXXXX" & "core.in:imuxsock.###" files are filling up /storage/core in VC

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• vCenter Server /storage/core partition is filled up with "core.rs:main Q:Reg.###" and "core.in:imuxsock.###"
• Space utilization on /storage/log partition also shows high due to too many stdout or stderr or *.backup files.

Cause

This issue is caused due to the “/storage/log” bloating. This is known issue has been already fixed in VC 7.0 U3i and VC 8.0U3 and higher versions which will prevent the issue from happening post applying the fix. However, the older log files and coredumps need to be manually cleaned up in the vCenter Server.

Resolution

Clean-up the files by following below steps :

1. Login to the VC using SSH/Putty session using root credentials.
   
   
2. Cleanup the log files using below commands:
   
   • find /var/log/vmware -type f -name "*.backup" | xargs -i rm -f {}
   • find /var/log -type f -name "lastlog.*.backup" | xargs -i rm -f {}
   • find /var/log -type f -name "tallylog.*.backup" | xargs -i rm -f {}
   • find /var/log/vmware/ -type f -mtime +7 -iname "*.stdout-*" | xargs -i rm -f {}
   • find /var/log/vmware/ -type f -mtime +7 -iname "*.stderr-*" | xargs -i rm -f {}
     
     
3. Cleanup the core-dump files using below commands:
   • find /storage/core/ -iname "core.in:imuxsock*" | xargs -i rm -f {}
   • find /storage/core/ -iname "core.in:imfile*" | xargs -i rm -f {}
   • find /storage/core/ -iname "core.rs:main Q:Reg*" | xargs -i rm -f {}
     
     
4. Stop the VC Services.
   • service-control --stop --all
     
     
5. Cleanup any logs older than 50 days.
   • find /var/log/vmware/ -type f -mtime +50 | xargs -i rm -f {}
     
     
6. Reload the config and restart rsyslog service
   • systemctl daemon-reload
   • systemctl restart rsyslog
     
     
7. Start all the VC services
   • service-control --start --all

Additional Information

To Identity if the current vCenter Server has the log bloating issue, take SSH/Putty session to VC and run the below commands,

• find /var/log/vmware/ -iname "*.stdout-*" | wc -l
• find /var/log/vmware/ -iname "*.stderr-*" | wc -l
• find /var/log/vmware/ -iname "messages.*.backup" |wc -l


Output for the 1st and 2nd command will be in thousands and 3rd command will be in hundreds if the VC is impacted by the log bloating issue and manual steps mentioned under Resolution is required to clean these files.

To Identify if the fix is available in VC using below methods

• Login to the vCenter Server using SSH/Putty session,
• Run the below command to see if the preventive fix is already available:
  
  
• cat /etc/cron.d/syslog.cron
  • If the output returns with "* * * * * root /usr/lib/applmgmt/syslog/scripts/syslog_cron.sh >/dev/null 2>&1" then the fix is available in the vCenter Server.
    
    
• cat /etc/logrotate.conf
  • If the output contains the keywords 'daily', 'size', 'rotate' and 'maxage' then the fix is available in the vCenter Server.