For several ESXi hosts in a cluster which are in a "Not responding" state, datastores are inaccessible (as seen in the vSphere client)

Products

• VMware vCenter Server

Issue/Introduction

The hostd process is found to be repeatedly crashing on the affected ESXi hosts.

It may not possible to log in to the affected ESXi hosts web UI or to SSH in to them.

The affected ESXi hosts may not reply to ping.

Rebooting an affected ESXi host does not resolve the issue, and may make the symptoms worse - for example: an affected host that was replying to ping no longer replies to ping.

Post rebooting an affected ESXi host it is observed that there are missing uplinks.

Environment

VCF 5.x

Cause

An underlying, potentially transient, storage issue may be responsible.

The ESXi hosts DVS configuration may be stored on the impacted storage, thus a storage issue could result in a situation where the dvsData.db copy on the host is not consistent with that on the vCenter.

The DVS configuration used by the ESXi host is most likely held in volatile memory and may have diverged from that held on the vCenter, rebooting an affected ESXi hosts means that the DVS configuration used by the ESXi host may diverge even more - resulting in the missing uplinks observed.

Resolution

Instead of rebooting an affected ESXi host as part of the troubleshooting process, this article should be implemented to remove the host from the DV switch and add the host back to the DV switch - 

https://knowledge.broadcom.com/external/article/385528

Once the article is implemented, hostd is expected to stabilize on the affected hosts and the issue is expected to be resolved.

Additional Information

The hostd crash dump back trace should be found to match the hostd crash dump back trace found in /var/core/ on the affected ESXi hosts.