The hostd service in ESXi 7.0U3 crashes due to memory corruption

Products

• VMware vSphere ESXi

Issue/Introduction

On ESXi 7.0U3, the hostd service crashes, with log messages similar to this are seen in the vobd.log file:

[YYYY-MM-DDTHH:MM:SS]: [UserWorldCorrelator] 304957116us: [esx.problem.hostd.core.dumped] /bin/hostd crashed (1 time(s) so far) and a core file may have been created at /var/core/hostd-zdump.000. This may have caused connections to the host to be dropped. [YYYY-MM-DDTHH:MM:SS]: An event (esx.problem.hostd.core.dumped) could not be sent immediately to hostd; queueing for retry. 
[YYYY-MM-DDTHH:MM:SS]: [UserWorldCorrelator] 309002531us: [vob.uw.core.dumped] /bin/hostd(2103800) /var/core/hostd-zdump.001 
[YYYY-MM-DDTHH:MM:SS]: [UserWorldCorrelator] 313055552us: [vob.uw.core.dumped] /bin/hostd(2103967) /var/core/hostd-zdump.002
[YYYY-MM-DDTHH:MM:SS]: [UserWorldCorrelator] 317109536us: [vob.uw.core.dumped] /bin/hostd(2104096) /var/core/hostd-zdump.003 
[YYYY-MM-DDTHH:MM:SS]: [UserWorldCorrelator] 321129801us: [vob.uw.core.dumped] /bin/hostd(2104225) /var/core/hostd-zdump.000

Environment

VMware vSphere 7.0.x

Cause

This hostd crash is caused by memory corruption in a function that is part of the time service event monitoring (enabled by default)

Resolution

This issue is resolved in ESXi 7.0 Update 3c.
The issue was also resolved in ESXi 7.0 U3b (No longer available)

Workaround 1:

To prevent hostd from crashing, disable time service event monitoring by following these directions:

• On vSphere Client, go to Configure -> System -> Time Configuration tab, select "Network Time Protocol" and click on EDIT button. 
  
  
  
  
•  From the configuration box, uncheck "Enable monitoring events"
  
  
  
  
• Click the OK button
  
  
  
  

Workaround 2:

If the impacted ESXi host is standalone or if the Host is in a Disconnected state in the vCenter due to hostd being in a down state, implement the following steps:

• From the SSH session of the impacted host, execute the following steps:
  
  • Check the current NTP configuration:
    configstorecli config current get -c esx -g system -k system_time


  • Create a temporary JSON file:
    vi /tmp/test.json


  • Add the following parameter in the JSON file:
    {
"events": false
}


  • Update the existing configuration with the new parameter:
    configstorecli config current update -c esx -g system -k system_time -infile /tmp/test.json


  • Start the hostd service:
    /etc/init.d/hostd start 


  • Ensure that the hostd service doesnt crash further and the status is Connected in the vCenter.