Error: No healthy upstream due to core.in:imuxsock core dumps filling /storage/core on vCenter Server
search cancel

Error: No healthy upstream due to core.in:imuxsock core dumps filling /storage/core on vCenter Server

book

Article ID: 377991

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • vCenter Server UI is inaccessible with the error: "No healthy upstream"

  • Multiple vCenter Server services fail to start when executing the command: service-control --start --all

  • The vCenter Server Management Interface (VAMI) at https://<vCenter_Server_FQDN/IP>:5480 displays a critical alert under Summary > Storage.

  • The /storage/core partition shows a utilization level of 90% or higher when verified using the command : df -h

  • Multiple core.in:imuxsock file reside within the /storage/core/ partition: find /storage/core/ -name "core.in:imuxsock*" -type f | wc -l

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue is due to the continuous generation of core.in:imuxsock.##### files triggered by a "too many open files" error within journalctl. The root cause is often inode exhaustion in the /var/log/vmware directory, occurring when the total number of log files exceeds the system's capacity. To confirm if log volume is contributing to the issue, verify the file counts in the following directories

  • find /var/log/vmware/applmgmt/ -iname "*.stdout-*" | wc -l

  • find /var/log/vmware/applmgmt/ -iname "*.stderr-*" | wc -l

  • ls /var/log/vmware/vsphere-ui/logs/ | wc -l

  • find /var/log/vmware/ -iname "messages.*.backup" | wc -l

If any of these counts reach thousand, the high log volume is causing the issue.

Resolution

Note: Prior to executing the following steps, take a backup or an offline snapshot (powered-off state) of the vCenter Server. If the vCenter Sever is part of an Enhanced Linked Mode (ELM) setup, simultaneous snapshots must be taken for all replicating vCenter Servers. Refer VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice 

  1. SSH into the vCenter Server as the root user.

  2. Navigate to the core directory and remove the imuxsock core dump files.

    1. cd /storage/core

    2. rm core.in:imuxsock.*

  3. If the core.in:imuxsock.#### files continue to generate, a more comprehensive cleanup of old log files is required:

    1. Delete the old backup files:

      1. find /var/log/vmware -type f -name "*.backup" | xargs -i rm -f {}

      2. find /var/log -type f -name "lastlog.*.backup" | xargs -i rm -f {}

      3. find /var/log -type f -name "tallylog.*.backup" | xargs -i rm -f {}

    2. Stop all vCenter Server services: service-control --stop --all

    3. Delete stdout/stderr files (older than 7 days):

      1. find /var/log/vmware/ -type f -mtime +7 -iname "*.stdout-*" | xargs -i rm -f {}

      2. find /var/log/vmware/ -type f -mtime +7 -iname "*.stderr-*" | xargs -i rm -f {}

    4. Delete rsyslog core dumps:

      1. find /storage/core/ -iname "core.in:imuxsock*" | xargs -i rm -f {}

      2. find /storage/core/ -iname "core.in:imfile*" | xargs -i rm -f {}

    5. Delete vSphere-ui and backup message files:

      1. rm -f /var/log/vmware/vsphere-ui/logs/vsphere-ui-gc.log.*

      2. rm -f /var/log/vmware/messages.*.backup

    6. Restart the rsyslog service: systemctl restart rsyslog

    7. Start all vCenter Server services: service-control --start --all

Additional Information

Powered by Wolken Software