ESXi hosts shows event /bin/hostd crashed, core file might have been created

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Some ESXi hosts can be seen in a disconnected state in vCenter Server, others in the same cluster may still be connected.
• ESXi version is earlier than 8.0U2
• The vSphere Client > Monitor > Events tab of the ESXi host may show warnings that "/bin/hostd crashed ..."
• The /var/run/log/hostd.log on the ESXi host shows a hostd crash(es) has occurred:

[YYYY-MM-DDTHH:MM:SS] In(166) Hostd[3756132]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 128 : /bin/hostd crashed (17 time(s) so far) and a core file might have been created at /var/core/hostd-zdump.000. This might have caused connections to the host to be dropped.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

• Checking the storage device plugins available on the ESXi host using the following command via ssh does not list both HPP and NMP (may show one only): 

vsish -e ls /storage/scsifw/plugins

Environment

VMware vSphere ESXi 8.0.1 and below

Cause

This issue can occur if either of the HPP or NMP module is not loaded. When telemetry data is collected on the ESXi host the NULL pointer resulting from the missing module causes hostd to crash.

Resolution

This issue is resolved in ESXi 8.0 Update 2

Some environments will see the issue stop with the following workaround applied:

• Disable the CEIP advanced setting on impacted ESXi hosts to see if this prevents the hostd crashes.
  From the vSphere Client select the ESXi host > Configure > Advanced System Settings, filter for Key UserVars.HostClientCEIPOptIn, if this is currently set to 1 then change it to 2.
  
  (Value indicates whether or not to opt-in for CEIP in Host Client, 0 for ask, 1 for yes, 2 for no)