ESXi host stops logging and eventually becomes unresponsive.

Products

• VMware vSphere ESXi

Issue/Introduction

• Post upgrade/update to ESXi 8.0.1 or later ESXi stops logging and eventually becomes unresponsive.
  
  
• The issue is observed in ESXi 7.0.1 or later as well.
  
  
• This issue is random in nature and there is no definitive way to identify which host is impacted by this issue.
  
  
• Hosts typically becomes unresponsive within 12 to 24 hours after logging stops.
  
  
• Logging is stopped across all the host logs like vmkernel.log, vobd.log, hostd.log, vmkwarning.log, vmksummary.log etc...

Environment

vSphere ESXi 7.X

vSphere ESXi 8.X

Cause

On reloading vmsyslogd a SIGHUP signal is sent to vmsyslogd. When multiple reload signal are sent with very short delay it could lead to a deadlock as the signal handler is not using async signal safe functions.

Resolution

Fix available in following release:

• VMware vSphere 7.0 P10 - https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/7-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-70u3v-release-notes.html 

• VMware vSphere 8.0 P05 - https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/release-notes/esxi-update-and-patch-release-notes/vsphere-esxi-80u3e-release-notes.html 

Workaround:

1. To workaround the issue before the host becomes unresponsive, execute pkill vmsyslogd. The watchdog will automatically restart the service, restoring functionality.

pkill vmsyslogd

2. If the host has already entered the unresponsive state, the only recovery option is to perform a reboot of the host.