VMs crashed due to esx.problem.vmfs.heartbeat.timedout issue

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESX 7.x

Issue/Introduction

Some VMs crashed, the following error message is found in vobd.log

YYYY-MM-DDThh:mm:ssZ In(14) vobd[#######]:  [UserWorldCorrelator] ###########us: [vob.uw.core.dumped] /bin/vmx(2484691) /vmfs/volumes/########-########-####-############/<Example-VM-name>/vmx-zdump.000
YYYY-MM-DDThh:mm:ssZ In(14) vobd[#######]:  [UserWorldCorrelator] ###########us: [esx.problem.application.core.dumped] An application (/bin/vmx) running on ESXi host has crashed (1 time(s) so far). A core file may have been created at /vmfs/volumes/########-########-####-############/<Example-VM-name>/vmx-zdump.000.

Environment

Esxi 7.0.x

Esxi 8.0.x

Cause

Storage access is extremely slow,  VMFS  heartbeat timed out:

 

Found the following issue in vobd.log: 

YYYY-MM-DDThh:mm:ss In(14) vobd[#######]:  [vmfsCorrelator] ############us: [esx.problem.vmfs.heartbeat.timedout] ########-########-####-############ <Example-datastore-name>

Resolution

This is a storage issue.

It can be  caused by a physical storage device issue, or an extremely bad performance of the storage.

Need to fix the storage issue first, then the VM crashing issue will  be fixed.