Frequent VMFS Heartbeat Recovered events reported by vCenter/ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• VMFS heartbeat recovered events are reported frequently in vCenter events in the vSphere Client
  
  
• These events are reported against multiple VMFS datastores 
  
  
• VMs operation or performance may be impacted

Environment

VMware vSphere vCenter (all versions)
VMware vSphere ESXi (all versions) 

Cause

• These events indicate that VMFS heartbeat timeout has occurred briefly and then datastore heartbeat has been recovered.
• ESXi /var/log/vobd.log reports entries similar to: 
  
  vobd[2098532]  The event ([esx.problem.vmfs.heartbeat.timedout] <VMFS volume UUID> <DatastoreName>) was sent immediately to hostd;
vobd[2098532]  [vmfsCorrelator] 14807596740549us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume <VMFS volume UUID> <DatastoreName>  [Timeout] [HB state abcdef02 offset 3866624 gen 4923 stampUS 14806948640524 uuid <VMFS volume UUID> jrnl <FB 41943053> drv 24.82]
  
  

These brief heartbeat timeouts can be triggered by a variety of issues storage issues, e.g.:

• intermittment I/O failures
• specifically intermittent I/O failures due to the array repeated reported devices busy (D:0x8)
• extreme I/O latency

Resolution

• Review ESXi logs for further information on the soruce of the I/O failures 
  
  
• Remediate the causes of the I/O failures. 

Additional Information

For more information, see Understanding lost access to volume messages in ESXi .