vSphere HA Failed to Failover VMs During Host Hardware Failure

Products

• VMware vCenter Server

Issue/Introduction

During a host hardware issue the affected virtual machines did not failover automatically to other healthy hosts.

The impacted VMs went into a Disconnected state and remained unreachable until manually restarting the failed host, after which the VMs moved to other hosts.

Environment

vCenter 8.x

Cause

Hardware failure on the host, FDM service not running.

• A time gap can bee seen in the hostd and vpxa logs on the Host.
• In the /var/run/log/fdm.log
• We see another time gap as to the last time the FDM service was running on the host. 

In the /var/run/log/vobd.log 

vobd.log
vob.user.hostd.host.poweroff.reason] Reason for power off: <Reason for reboot: The host is being powered off through host
d.>, User: <vpxuser:VSPHERE.LOCAL\Administrator>
s: [vob.user.hostd.host.poweroff.reason] Reason for power off: <Reason for reboot: The host is being powered off through ho
std.>, User: <vpxuser:VSPHERE.LOCAL\Administrator>
 [esx.audit.hostd.host.poweroff.reason] The host is being powered off through hostd. Reason for power off: The host is be
ing powered off through hostd.. User: vpxuser:VSPHERE.LOCAL\Administrator. Please consult ESXi Embedded and vCenter Server Setup Guide or follow the Ask VMware link for more information.
vobd[0000]:  [GenericCorrelator] 00000000us: [vob.user.host.stop.shutdown] Host is shutting down.

Resolution

Reboot the affected Host. 

Additional Information

This is a rare occurrence.