Virtual machine becomes inaccessible or hangs after a power outage or loss of connectivity to the storage

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
After experiencing a host reboot due to a power outage or lost of connectivity to the shared storage, you experience these symptoms:

• Virtual machine becomes inaccessible or hangs.
• In the /var/log/vmkernel.log file of the ESXi host, you see entries similar to:
  
  WARNING: HBX: XXXX: Replay of journal <type 6 addr XXXXXXXX> on vol 'VOLUME_NAME' failed: Lost previously held disk lock.

Cause

VMFS is a journaling file system where it keeps track of changes not committed to disk yet. These journals are later replayed by the hosts.

In a rare scenario, one can have a situation where two hosts that share a common resource tries to replay a given journal in parallel. This can cause the journal not to be completely replayed and can cause certain locks (taken on resource) to be lost. This in turn can cause a VM on the host to get hung or inaccessible.

Note: This issue is specific to VMFS-6 only and can be seen with vSphere 6.5 and above release with VMFS-6.

Resolution

This is a known issue affecting VMware ESXi 6.5.x and 6.7.x.

This has been fixed in ESXi 6.5 Update 3 and 6.7 Update 2. Please refer VMware vSphere downloads, VMware Converter, OEM custom images, patches and addons in the Broadcom Support Portal to download patches.


Workaround:
Please follow the article, "Investigating virtual machine file locks on ESXi" to find and resolve the file lock".