HA failover over of virtual machines on an ESXi host with "Lost connectivity to datastore" warnings

Products

• VMware vSphere ESXi

Issue/Introduction

• A HA failover is triggered on VMs on an ESXi host
• The HA master node reports lost access to the host /var/log/fdm.log on HA master node:
  

  Fdm[2108683] [Originator@6876 sub=Cluster opID=clusterManager.cpp:980-601c5b31] Marking slave host-####### as unreachable

• The impacted host stops logging for a number of minutes at the time of the failover. (All logs under /var/log show a gap in logging)
• Prior to stopping logging, the ESXi host does not report datastore heartbeat timeout
• On resuming logging, the ESXi host /var/log/vobd.log reports datastore heartbeat timeout, and then recovers heartbeat within one to two seconds:
  

  vobd[2098147]  [vmfsCorrelator] 391182884031us: [vob.vmfs.heartbeat.timedout] <Datastore UUID>  <Datastore Name>
  vobd[2098147]  [vmfsCorrelator] 391184193077us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume  <Datastore UUID> (<Datastore Name>): [Timeout] [HB state abcdef02 offset 3837952 gen 81 stampUS 391184176867 uuid <UUID>  jrnl <FB 25165830> drv 24.82]

Environment

VMware vSphere ESXi (all versions)

Cause

HA failover occurs when an ESXi host becomes unresponsive. The heartbeat timeouts and lost access to datastores can be a cause of the host unresponsiveness, which subsequently triggers HA failover.

While underlying hardware issues (such as faulty components) are a common cause of this behavior, it is equally important to consider environmental factors. Transient network instability, such as network flapping or switch issues, can induce temporary loss of datastore connectivity (e.g., All Paths Down - APD conditions), leading to host unresponsiveness and triggering unnecessary vSphere HA recovery workflows.

Thorough troubleshooting should include both physical hardware diagnostics and an analysis of the network fabric connectivity to the storage to isolate the trigger of the connectivity loss.

Resolution

• Review host logs (/var/log/fdm.log, /var/log/vobd.log, and /var/log/vmkwarning.log) to correlate timestamps of HA failovers with storage or network events.
• For Hardware issues: If logs indicate physical hardware warnings (e.g., SCSI controller errors, PCIe errors), engage the hardware vendor for diagnostics.
• For Environmental/Network issues: If logs indicate network instability, link state changes, or transient APD conditions, investigate the physical network switch, cabling, and uplink configuration.
• Ensure ESXi host firmware and drivers are up to date according to the hardware vendor's HCL.