Troubleshoot ESXi heartbeat datastore problems affect vSphere HA

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Provide information on how to troubleshoot and fix vSphere HA problems due to heartbeat datastore problems

Symptoms:

• vSphere HA will not configure on one or more ESXi hosts in a cluster
• Datastores used for vSphere HA heartbeat have changed or not yet been configured

Environment

VMware vCenter Server 8.0
VMware vCenter Server 7.0
VMware vCenter Server 6.0

Cause

For ESXi hosts which were working with vSphere HA previously the problem is most often that the previous heartbeat datastore was deprecated or has become not available to all the ESXi hosts in the cluster.
For ESXi hosts which are being newly configured for vSphere HA, the heartbeat datastore has not yet been selected

Resolution

For vSAN ESXi host clusters:

Ensure the host(s) are networked to, configured for, and participating in the vSAN cluster. ESXi host(s) cannot use the vSAN datastore as a heartbeat datastore. 
If it is non-vSAN datastores(VMFS or NFS) to use for a heartbeat datastore:

1. Ensure the isolation response is set to power-off and restart VMs.
2. Set an isolation address to be on the same network as the vSAN network.
3. Follow Setting Multiple Isolation Response Addresses for VMware High Availability.

Otherwise, follow the non-vSAN instructions, if you do have vSAN hosts with access to VMFS/NFS datastores for heart beating.

For non-vSAN ESXi host clusters:

Find out which datastores are being used for vSphere HA heart beating.

1. Select the cluster the affected ESXi host(s) are in, select Configure > vSphere Availability > EDIT.
2. Select Heartbeat Datastores. Note which datastore(s) are available, select Cancel.
3. Select the problem ESXi host > Datastores.
4. Note whether the heartbeat datastore(s) you found earlier show on the list of datastores available on the affected ESXi host.

If this host is able to see the heartbeat datastore(s) listed, note whether you see any error icon such as the red exclamation mark.

1. If you see red exclamation mark, click on the name of the affected datastore to see its Summary page. Note whether the error shown is likely to be relevant -- if so, you need to either:
   • Change the heartbeat datastore(s) for vSphere HA at the cluster level. Follow Configure Heartbeat Datastores.
   • Troubleshoot and fix the issue with the existing heartbeat datastore(s). Look for detailed errors in:
     • vmkwarning.log
     • vmkernel.log
     • vobd.log
     • hostd.log
2. You can further verify what the ESXi host thinks its heartbeat datastore is at the command line:

The following command shows what the ESXi host has marked as its heartbeat datastore:

for uuid in $(lsof |awk '$2=="fdm" ' |grep FILE |grep -v /var/log |awk 'BEGIN { FS = "/" }; {print $4}'); do localcli storage filesystem list |grep $uuid| awk '{print $1, $2}';done;
 

The following command will show what the ESXi host has logged in the fdm.log as the heartbeat datastore. This command will also show you some previous datastores used, if any:

for uuid in $(cat /var/log/fdm.log |grep "Creating heartbeat file" |tail -n 10 |awk 'BEGIN { FS = "/" }; {print $4}' |awk '{print $1}' |sort |uniq); do localcli storage filesystem list |grep $uuid| awk '{print $1, $2}';done;

Additional Information

Troubleshooting Heartbeat Datastores
vSphere Availability
About vSphere Storage
Handling Failures and Troubleshooting vSAN
vVols Datastore Is Inaccessible

Impact/Risks:
vSphere HA will not configure if the ESXi host is not using a valid heartbeat datastore