ESXi host stops logging and eventually becomes unresponsive.

Products

• VMware vSphere ESXi

Issue/Introduction

• Post upgrade/update to ESXi 8.0.1 or later ESXi stops logging and eventually becomes unresponsive.
• The issue is observed in ESXi 7.0.1 or later as well.
• This issue is random in nature and there is no definitive way to identify which host is impacted by this issue.
• Hosts typically becomes unresponsive within 12 to 24 hours after logging stops.
• Logging is stopped across all the host logs like vmkernel.log, vobd.log, hostd.log, vmkwarning.log, vmksummary.log etc...
• The following error in the hidden log .vmsyslogd.err located in /var/log/. Since this is a hidden log can be observed, it will not be visible with a standard ls command. To view it, please use ls -la within the /var/log/ directory to list all files, including hidden ones. The initial error/switch will coincide with the time of logging stopping.:

  YYYY-MM-DDTHH:MM vmsyslog : CRITICAL] vmsyslogd daemon starting (#####)
YYYY-MM-DDTHH:MM vmsyslog.main : ERROR ] switching to normal mode from init mode
YYYY-MM-DDTHH:MM vmsyslog.main : CRITICAL] Dropping messages due to log stress (qsize = 25000)

Environment

vSphere ESXi 7.X
vSphere ESXi 8.X

Cause

On reloading vmsyslogd a SIGHUP signal is sent to vmsyslogd. When multiple reload signal are sent with very short delay it could lead to a deadlock as the signal handler is not using async signal safe functions.

Resolution

Fix available in following release:

• VMware vSphere 7.0 P10 - https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-70u3v-release-notes.html 

• VMware vSphere 8.0 P05 - https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3e-release-notes.html 

Workaround:

1. To workaround the issue before the host becomes unresponsive, execute pkill vmsyslogd. The watchdog will automatically restart the service, restoring functionality.
Note - If the Logging of the ESXi Host has already stopped which means that the ESXi host vmsyslogd is already in deadlock state, then the below command will not help recover the host.

pkill vmsyslogd

2. If the host has already entered the unresponsive state, the only recovery option is to perform a reboot of the host.

Additional Information

ESXi ホストのログ記録が停止し、最終的に応答しなくなる